reasonable handling of nullable values

What version of Go are you using (go version)?

$ go version
go version go1.15.6 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

https://play.golang.org/p/7kgBsW80Mzf

What is the problem?

If we would have not empty decoder.errorContext.FieldStack and we received error of fmt.wrapError type from UnmarshalJSON then later addErrorContext wouldn't be able to set proper value for err.Field and err.Struct because error type won't match with *UnmarshalTypeError.

https://github.com/golang/go/blob/a7e16abb22f1b249d2691b32a5d20206282898f2/src/encoding/json/decode.go#L181-L183 https://github.com/golang/go/blob/a7e16abb22f1b249d2691b32a5d20206282898f2/src/encoding/json/decode.go#L255-L258

PR https://github.com/guregu/null/pull/6 seems to have stalled, so here's another take on it.

Key points/thoughts/questions:

• No dependency on pq.
• One additional type: null.Time. If the null.Time version feels right, I'll add the zero.Time version.
• Uses RFC3339 time format just as the native time package does here.
• Unmarshals both string and time.Time type. I'm tempted to only unmarshal strings. Thoughts?

Like this library, very clean & nice to use.

Would you be interested in adding support for / accepting a PR with an implementation of null.Duration ?

Hello Greg (hope I got the name right :) )

First of all, great library for marshalling and unmarshalling Golang's SQL null datatypes. Really saved me a lot of time and effort.

However, I noticed that this library is missing one piece of functionality which is storing null values in database. I have gone through the library and could not find it anywhere. I had to implement this part in my own codebase. I would be happy to raise a Pull Request so that this functionality can be merged to master.

Please do let me know if I can go ahead with raising a pull request.

Testing my models on GraphQL, I detected some issues regarding text-marshaling implementation in this library.

null.Time returns "null"

func (t Time) MarshalText() ([]byte, error) {
	if !t.Valid {
		return []byte("null"), nil
	}
	return t.Time.MarshalText()
}

Other null.X implementations

// MarshalText implements encoding.TextMarshaler.
// It will encode a blank string when this String is null.
func (s String) MarshalText() ([]byte, error) {
	if !s.Valid {
		return []byte{}, nil
	}
	return []byte(s.String), nil
}

// MarshalText implements encoding.TextMarshaler.
// It will encode a blank string if this Int is null.
func (i Int) MarshalText() ([]byte, error) {
	if !i.Valid {
		return []byte{}, nil
	}
	return []byte(strconv.FormatInt(i.Int64, 10)), nil
}

My question/curiosity is: why does null.Time text-marshaling return a "null" string instead of an empty string like the others?

type JsonUser struct { ID null.Intjson:"id" db:"id"Email null.Stringjson:"email" db:"email"Password null.Stringjson:"-" db:"password"FirstName null.Stringjson:"first_name" db:"first_name"LastName null.Stringjson:"last_name" db:"last_name"CreatedAt null.Timejson:"created_at" db:"created_at"UpdatedAt null.Timejson:"updated_at" db:"updated_at"` }

func UserByID(w http.ResponseWriter, req *http.Request) { vars := mux.Vars(req) if userID, err := strconv.Atoi(vars["id"]); err != nil { fmt.Fprint(w, response.Success("No record found.", http.StatusNotFound, nil).JSON()) return } else { jsonUser := &JsonUser{} err = DB.QueryRow("SELECT id, email, first_name, last_name, created_at, updated_at FROM users WHERE id = ? LIMIT 1;", userID).Scan(&jsonUser.ID, &jsonUser.Email, &jsonUser.FirstName, &jsonUser.LastName, &jsonUser.CreatedAt, &jsonUser.UpdatedAt) if err != nil { fmt.Fprint(w, response.Error(err.Error(), http.StatusInternalServerError).JSON()) return } fmt.Fprint(w, response.Success("Request successful!", http.StatusOK, jsonUser).JSON()) }

return

}`

CreatedAt is not null, but UpdatedAt is for the particular record I'm retrieving.

The error that is being thrown is: "sql: Scan error on column index 4: null: cannot scan type []uint8 into null.Time: [50 48 49 56 45 48 56 45 50 50 32 48 48 58 53 54 58 50 50]"

It doesn't seem that this package's Time is not handling []uint8 but I'm not sure how to get this working. Thanks!

This adds the UnmarshalParam() function from the BindUnmarshaler interface from echo: https://github.com/labstack/echo/blob/60072188355b8972c738a041666bd4b05957c135/bind.go#L24

This is useful for unmarshaling form and query values, and since it's (mostly) a proxy to UnmarshalText() I don't see any downsides in adding it. The only exception is the Bool; I had to add on and off. I could add these just for UnmarshalParam, but it seemed useful to me to add it to UnmarshalText as well.

• Add Scan and Value methods for non sql.Int64 and sql.Float64 types
• Change NullInt to use int in the code opposed to int64
• Rename Float to Float64
• Add support for all missing key int and float types: float32, int8, int16, int32, int64, uint8, uint16, uint32, uint64
• Update README to include new additions/changes

Purpose: to allow cases where field it's value does not exist inside an wrapping object to be sent to the server as NULL JSON/JSONB Value (where structure type is json.RawMessage)

If someone could implement it it would be even better.

Unmarshalling and marshalling types in the null package isn't symmetric. For example:

type T struct {
  Name null.String
}

in := []byte(`{"Name":""}`)
var t T
json.Unmarshal(in, &t)
fmt.Printf("%+v\n", t) // => {Name:{NullString:{String: Valid:false}}}
out, _ := json.Marshal(t)
fmt.Println(string(out)) // => {"Name":null}

Why not have in unmarshal to {Name:{NullString:{String: Valid:true}}}? In addition to clearing up the issue shown above it would make the null package incredibly useful for the common issue of how to determine if a RESTful request intends to set a value to the zero value or if it just wasn't included in the request (a PATCH-like request).

Right now the common recommendation is to use pointers for the fields in your structs as described here, but I'd much prefer using the null package for this if I could.

Let me know what you think. I'm happy to create a PR for this if you think this idea has legs.

Hi guregu, My teammates and I really like your library, as it has helped us a lot when going from JSON to go structs to databases (thank you!). One small issue we found is unmarshalling large numbers from JSON. Right now, because there is only one number type in Javascript, integers in JSON that are large enough are decoded in scientific notation. Because of this, these numbers cannot be unmarshalled into integer types, in particular your null.Int and zero.Int types. I was wondering if you had any thoughts on the issue.

If you're interested, I've already written an implementation that passes all of the tests, if you want to take a look.

1. defind web api dto struct

type Dto struct {
   Start null.Time
}

2. send this json to web api

{ 
  "Start": null
}

4. service: use gorm update entry

gorm.Model(&MyTable).Where("id = 1").Update(&dto)

5. Hope to generate sql

update my_table set start = null  where id = 1

The start field is now ignored when used in this way

Hi @guregu,

I added a new method which checks if a string is zero or consists only of spaces. This is a helpful check especially for mandatory fields.

Would it be possible to integrate this into the null library? Otherwise I would have to release my own version. I think this could be interesting for others as well.

Regards. Nils

Any ideas or opinions on how to handle generics in this library? I played around with creating a generic Null[T] type, but I wonder if it's even useful at that point, because we are essentially recreating the Optional type you might find in other languages. At what point is it going too far? Maybe ideally we could make generic Null/Zero types and use the materialized version of it for each pre-existing type.

I'm also curious if we'll see a generic sql.Null[T] type in the standard library, which might obsolete a good chunk of this library, especially if this change lands: https://github.com/golang/go/issues/5901 (which lets you easily specify custom marshalers for JSON, moving the responsibility from the object itself to the encoding site).

See also: https://github.com/golang/go/issues/48702

Less of an issue, and more of an FYI. I have been keeping tabs on the (un)marshaling/(de)encoding issue with omitempty, and it looks like they have a adjacent proposal that may provide a workable solution:

https://github.com/golang/go/issues/5901

Didn't see anywhere else to leave a comment about this, and I figured it is relevant to this project in particular (I know its relevant to my usage of this project! Thanks by the way!)

Each struct of this library uses sql package (persistence or infrastructure layer), while it can be used for JSON marshaling (presentation layer). It seems to be against some software architecture patterns based on layered architecture or etc.

Adds Int from sql.NullString (super helpful when dealing with id's on databases where the id might not be in a struct used to receive/pass to client). It's helpful for my purposes, though there may be a better way you've found.