This adds new validation in UnmarshalText and Parse that returns an ErrInvalidCharacters error when called with input bytes outside the expected Base32 encoding set. See https://github.com/oklog/ulid/issues/21 for details.

This does unfortunately end up being a bit slower. Some basic benchmarks from my laptop:

Before

BenchmarkUnmarshal/Text-8         	1000000000	        17.3 ns/op	1506.08 MB/s

With a simple loop

BenchmarkUnmarshal/Text-8         	500000000	        32.5 ns/op	 799.66 MB/s

With a manually unrolled loop (this change)

BenchmarkUnmarshal/Text-8         	500000000	        26.6 ns/op	 976.36 MB/s

Comparison from original to this version

benchmark                       old ns/op     new ns/op     delta
BenchmarkUnmarshal/Text-8       17.3          26.6          +53.76%

benchmark                       old MB/s     new MB/s     speedup
BenchmarkUnmarshal/Text-8       1506.08      976.36       0.65x

Fixes https://github.com/oklog/ulid/issues/21.

I'm adding to the DB in very quick succession around 1000 rows.

I decided to output the ULIDs so I could catch if there were dupes:

Here are the relevant parts:

01F9VG0NT0ACNSHJM2J4G6WH3E
01F9VG0NT0ACNSHJM2J8WRTTJE
01F9VG0NT0ACNSHJM2JAC9W6YT
01F9VG0NT0ACNSHJM2JDWKQS65
01F9VG0NT0ACNSHJM2J8KV2SB2
01F9VG0NT0ACNSHJM2J4G6WH3E
Error 1062: Duplicate entry '01F9VG0NT0ACNSHJM2J4G6WH3E' for key 'PRIMARY'

01F9VG0NT0ACNSHJM2N53XTXJH
01F9VG0NT0ACNSHJM2N8BWJXPF
01F9VG0NT0ACNSHJM2N8QXRBDM
01F9VG0NT0ACNSHJM2N53XTXJH
Error 1062: Duplicate entry '01F9VG0NT0ACNSHJM2N53XTXJH' for key 'PRIMARY'

01F9VG0NT0ACNSHJM2SMR13HZ2
01F9VG0NT0ACNSHJM2SQVZW987
01F9VG0NT0ACNSHJM2SVQJ87QQ
01F9VG0NT0ACNSHJM2SMR13HZ2
Error 1062: Duplicate entry '01F9VG0NT0ACNSHJM2SMR13HZ2' for key 'PRIMARY'

01F9VG0NT0ACNSHJM30062WA6F
01F9VG0NT0ACNSHJM30297Z6DR
01F9VG0NT0ACNSHJM30062WA6F
Error 1062: Duplicate entry '01F9VG0NT0ACNSHJM30062WA6F' for key 'PRIMARY'

01F9VG0NT0ACNSHJM39T0D5B1B
01F9VG0NT0ACNSHJM39ZVD159R
01F9VG0NT0ACNSHJM3A59EMVZC
01F9VG0NT0ACNSHJM39XSN0A84
01F9VG0NT0ACNSHJM3A34FHCAR
01F9VG0NT0ACNSHJM3A5CKZKJT
01F9VG0NT0ACNSHJM39T0D5B1B
Error 1062: Duplicate entry '01F9VG0NT0ACNSHJM39T0D5B1B' for key 'PRIMARY'

01F9VG0NT0ACNSHJM3APRCBXA7
01F9VG0NT0ACNSHJM3ATGM0XHQ
01F9VG0NT0ACNSHJM3APRCBXA7
Error 1062: Duplicate entry '01F9VG0NT0ACNSHJM3APRCBXA7' for key 'PRIMARY'

01F9VG0NT0ACNSHJM3CTT1B09C
01F9VG0NT0ACNSHJM3CZHHK5QT
01F9VG0NT0ACNSHJM3D12FM3C4
01F9VG0NT0ACNSHJM3CXQGM6GM
01F9VG0NT0ACNSHJM3D3H1M9EE
01F9VG0NT0ACNSHJM3D4REME36
01F9VG0NT0ACNSHJM3CTT1B09C
Error 1062: Duplicate entry '01F9VG0NT0ACNSHJM3CTT1B09C' for key 'PRIMARY'

01F9VG0NT0ACNSHJM3DPF00J7M
01F9VG0NT0ACNSHJM3DG1C0KAC
01F9VG0NT0ACNSHJM3DQR5G89S
01F9VG0NT0ACNSHJM3DV3SHQ6M
01F9VG0NT0ACNSHJM3DY6049MW
01F9VG0NT0ACNSHJM3DPF00J7M
Error 1062: Duplicate entry '01F9VG0NT0ACNSHJM3DPF00J7M' for key 'PRIMARY'

01F9VG0NT0ACNSHJM3VYDT1FKJ
01F9VG0NT0ACNSHJM3VYDT1FKJ
Error 1062: Duplicate entry '01F9VG0NT0ACNSHJM3VYDT1FKJ' for key 'PRIMARY'

My code is broken into three parts, as it's a complex long running process:

stored in a struct:

now := time.Now().UTC()
entropy := ulid.Monotonic(rand.New(rand.NewSource(now.UnixNano())), 0)

I generate the ID (m is a struct):

job.ID = common.GenUlid(m.now, m.entropy)

Func:

func GenUlid(now time.Time, entropy io.Reader) string {
	return ulid.MustNew(ulid.Timestamp(now), entropy).String()
}

Thanks for your time.

type Team struct {
	ID         ulid.ULID `db:"id"`
	Name       string    `db:"name"`
	Status     string    `db:"status"`
	CreatedAt  time.Time `db:"created_at"`
	UpdatedAt  time.Time `db:"updated_at"`
	Archivedat time.Time `db:"archived_at"`
}

Usage:

	query := "SELECT * FROM teams WHERE status = 'active'"
	teams := []Team{}
	if err := r.DbConn.SelectContext(ctx, &teams, query); err != nil {
		return nil, errors.Wrap(err, "selecting teams")
	}

Result: error retrieving data selecting teams: sql: Scan error on column index 0, name "id": ulid: bad data size when unmarshaling

Table Structure:

create table teams (
        id varchar(26) not null,
        name varchar(255) not null,
        status varchar(26) not null,
        created_at datetime not null default now(),
        updated_at datetime not null default now(),
        archived_at datetime null
      ) default character set utf8mb4 engine = InnoDB;

This implements the method described by @alizain (and used in his ulid package) to enforce monotonicity of ULIDs within a same millisecond.

I'm not sure if the API is to your liking, I'm happy to reshape it into something aesthetically pleasing to you.

Addresses issue #6.

r: @peterbourgon @tsenart cc: @alizain

These are methods like Parse and MustParse, but validate that their inputs contain only valid base32 characters. On invalid input, they return a new ErrInvalidCharacters error rather than producing an undefined ULID.

A comparison benchmark:

$ go test -v -run=^$ -benchtime 10s -bench BenchmarkParse ./...
goos: darwin
goarch: amd64
pkg: github.com/oklog/ulid
BenchmarkParse-8         	500000000	        26.1 ns/op	 994.49 MB/s
BenchmarkParseStrict-8   	500000000	        39.2 ns/op	 662.87 MB/s
PASS
ok  	github.com/oklog/ulid	39.221s
?   	github.com/oklog/ulid/cmd/ulid	[no test files]

This is an alternate fix for https://github.com/oklog/ulid/issues/21. I originally approached this issue by adding this validation to Parse and UnmarshalText (https://github.com/oklog/ulid/pull/22), but there was some concern about the performance impact.

Fixes https://github.com/oklog/ulid/issues/21.

Are you publishing a standalone executable for the go version? Someone asked over on my repo, and I wanted to send them your way, because doing it with go would be pretty straightforward.

Proposing to integrate continuous fuzzing. Fuzzit gives free service for open source projects.

This targets Parse and ParseStrict by input + New by fuzzing the timestamp and entropy.

Setup is like this:

• In Fuzzit create targets ulid-new ulid-parse ulid-parse-strict.
• In Fuzzit settings grab an API key. On the repo settings in Travis paste it to envvar FUZZIT_API_KEY.

Thanks for considering.

@peterbourgon @tsenart

I'm using this library to generate ULID's for a primary key in a database and when I run my benchmarks I'm finding a lot of duplicate primary key errors from my database. I'm assuming I'm using this library wrong because. What is the guidance on how to use this to avoid generating duplicate keys?

I'm doing the following.

now := time.Now()
entropy := ulid.Monotonic(rand.New(rand.NewSource(now.UnixNano())), 0)

transaction, err := pool.Begin()

for i := 0; i < b.N; i++ {
    id := ulid.MustNew(ulid.Timestamp(now), entropy)
    
    // INSERT SQL
    tx.Exec(INSERT_SQL_USING_id)
}

tx.Commit()

What have I done wrong that generates duplicate ULID's?

Has there been any consideration into allowing string UUIDs be parsed within the unexported parse function: func parse(v []byte, strict bool, id *ULID) error

The reason I'm raising this is because I'd like to use this library for my primary keys, however CockroachDB returns UUID columns as UUID strings, not []byte, so ulid.ULID#Scan(src) fails to parse the value.

My workaround is to implement a wrapper ULID type which has a Scan function (implementation shown below) however it would be nice to not have to do this, as I have to do things like ULID{ULID: ulid.MustParse(...)} which is a real annoyance:

func (id *ULID) Scan(src interface{}) error {
	if str, ok := src.(string); ok {
		if len(str) != ulid.EncodedSize {
			bytes, err := uuid.Parse(str)
			if err != nil {
				return err
			}

			return id.UnmarshalBinary(bytes[:])
		}
	}

	return id.ULID.Scan(src)
}

If you don't want to implement UUID parsing or add an extra dependency, how about some mechanism where users can register custom parsers?

Hi,

I've been experimenting with your library to generate ULIDs as keys for a database. My code was doing 20+ row insertions per 1ms, and results, i.e. row ULIDs, are not in a strict sequence, not sorted (image below). I was suggested that this may be due to "Within the same millisecond, sort order is not guaranteed" on https://github.com/alizain/ulid which, as I understand, served as a basis for oklog/ulid.

Could you please confirm (or not) randomness within 1 ms? If not, below is my code, I would appreciate your check if IDs are generated correctly:

func GetNewID() string {
	entropy := rand.New(rand.NewSource(time.Now().UnixNano()))
	value := ulid.MustNew(ulid.Now(), entropy)
	return value.String()
}

This is a select in descending order by id (generated by ULID), where post_id column is filled with a "for" loop key. With correct generation, post_id should be a in a strict desc. order.

Thank you! D.

var list = make([]string, 0) var monotonic = ulid.Monotonic(rand.New(rand.NewSource(time.Now().UnixNano())), 1) var busy = make(chan bool, 1)

func AddUlid2List(index string) { busy <- true id, _ := ulid.New(ulid.Timestamp(time.Now()), monotonic) list = append(list, id.String()+" "+index) <-busy }

func Test_test6(t *testing.T) { wg := sync.WaitGroup{} wg.Add(2) go func() { for i := 0; i < 1000000; i++ { AddUlid2List("1") } wg.Done() }() go func() { for i := 0; i < 1000000; i++ { AddUlid2List("2") } wg.Done() }() wg.Wait() for i := 0; i < len(list); i++ { if i+1 < len(list) && list[i] > list[i+1] { //63718 01GH13A2V6MBDBNBG8TSH0B0ZD 2 01GH13A2V5W3TXG0EX5GF9Q4MP 1 fmt.Println(i, list[i], list[i+1]) t.Error("big bug") } } if list[len(list)-2] > list[len(list)-1] { t.Error("big bug2") fmt.Println(list[len(list)-2], list[len(list)-1]) } fmt.Println("END") }

func Test_ulid(t *testing.T) { i := 0 for { i++ Test_test6(t) if i == 10 { break } } }

run the method Test_ulid, it print: 63718 01GH13A2V6MBDBNBG8TSH0B0ZD 2 01GH13A2V5W3TXG0EX5GF9Q4MP 1 big bug

Hi! I'm writing a new Ruby library for handling ULID in these days. Now I’m testing other implementations examples in https://github.com/kachick/ruby-ulid/issues/53.

And I have found weird examples in original repository as https://github.com/ulid/javascript/pull/85.

And then checked the parser of this library, because I'm using this in a Go project, it is so useful! 😄

Using this command line tool as below, the version is https://github.com/oklog/ulid/tree/e7ac4de44d238ff4707cc84b9c98ae471f31e2d1

$ ulid -h
Usage: ulid [-hlqz] [-f <format>] [parameters ...]
 -f, --format=<format>  when parsing, show times in this format: default, rfc3339, unix, ms
 -h, --help             print this help text
 -l, --local            when parsing, show local time instead of UTC
 -q, --quick            when generating, use non-crypto-grade entropy
 -z, --zero             when generating, fix entropy to all-zeroes

$ ulid 01111111111111111111111111
Mon Dec 19 08:09:04.801 UTC 2005

$ ulid 0LLLLLLLLLLLLLLLLLLLLLLLLL # `L` is same as `1` in https://www.crockford.com/base32.html, but returned different value
Tue Aug 02 05:31:50.655 UTC 10889

$ ulid 0UUUUUUUUUUUUUUUUUUUUUUUUU # `U` is invalid in https://www.crockford.com/base32.html, but does not raise error
Tue Aug 02 05:31:50.655 UTC 10889

$ ulid 00000000000000000000000000
Thu Jan 01 00:00:00 UTC 1970

$ ulid 0OOOOOOOOOOOOOOOOOOOOOOOOO # `O` is same as `0` in https://www.crockford.com/base32.html, but returned different value
Tue Aug 02 05:31:50.655 UTC 10889

In my understanding, Crockford's base32 does not contain L I O for the encoded product. So I think ULID can handle them as invalid values 🤔 ref: https://github.com/ulid/spec/issues/38, https://github.com/kachick/ruby-ulid/issues/57