Vitess is a database clustering system for horizontal scaling of MySQL.

Overview

Maven Central Build Status codebeat badge Go Report Card FOSSA Status CII Best Practices Coverage

Vitess

Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding.

By encapsulating shard-routing logic, Vitess allows application code and database queries to remain agnostic to the distribution of data onto multiple shards. With Vitess, you can even split and merge shards as your needs grow, with an atomic cutover step that takes only a few seconds.

Vitess has been a core component of YouTube's database infrastructure since 2011, and has grown to encompass tens of thousands of MySQL nodes.

For more about Vitess, please visit vitess.io.

Vitess has a growing community. You can view the list of adopters here.

Reporting a Problem, Issue ,or Bug

To report a problem the best way to get attention is to create a GitHub issue.

For topics that are better discussed live, please join the Vitess Slack workspace. You may post any questions on the #general channel or join some of the special-interest channels.

Follow Vitess Blog for low-frequency updates like new features and releases.

Security

Reporting Security Vulnerabilities

To report a security vulnerability, please email vitess-maintainers.

See Security for a full outline of the security process.

Security Audit

A third party security audit was performed by Cure53. You can see the full report here.

License

Unless otherwise noted, the Vitess source files are distributed under the Apache Version 2.0 license found in the LICENSE file.

FOSSA Status

Issues
  • workflow: Add horizontal resharding workflow.

    workflow: Add horizontal resharding workflow.

    This is a beta version implementation, which simply supports creating the horizontal workflow through UI and updating progress of each step.

    In the end-to-end test (test/workflow_horizontal_resharding.py), only happy path is tested. To simplify the test, rather than fetching information from front-end and checking explicitly, I manually interacting with the UI after setting up the environments.

    Unit test will be added later.

    Possible Optimizations (wait for feedback from reviewers, the listed points might be useless):

    • make UI more friendly, s.t. the user input doesn't have to input the shard list
    • supports action for the workflow, which allows the user to stop and restart the workflow
    • adding possible corner case end-to-end test

    This change is Reviewable

    opened by wangyipei01 87
  • Experimental: automated, scheduled, dependency free online DDL via gh-ost/pt-online-schema-change

    Experimental: automated, scheduled, dependency free online DDL via gh-ost/pt-online-schema-change

    This PR (work in progress) introduces zero dependency online schema changes with gh-ost/pt-online-schema-change.

    UPDATE: this comment edited to reflect support for pt-online-schema-change. Originally this PR only supported gh-ost. Mostly whenever you see gh-ost, consider pt-online-schema-change to apply, as well.

    TL;DR

    User will issue:

    alter with 'gh-ost' table example modify id bigint not null;
    
    alter with 'pt-osc' table example modify id bigint not null
    

    or

    $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global \
        ApplySchema -sql "alter with 'gh-ost' table example modify id bigint unsigned not null" commerce
    
    $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global \
        ApplySchema -sql "alter with 'pt-osc' table example modify id bigint unsigned not null" commerce
    

    and vitess will schedule an online schema change operation to run on all relevant shards, then proceed to apply the change via gh-ost on all shards.

    While this PR is WIP, this flow works. More breakdown to follow, indicating what's been done and what's still missing.

    The ALTER TABLE problem

    First, to iterate the problem: schema changes have always been a problem with MySQL; a straight ALTER is a blocking operation; a ONLINE ALTER is only "online" on the master/primary, but is effectively blocking on replicas. Online schema change tools like pt-online-schema-change and gh-ost overcome these limitations by emulating an ALTER on a "ghost" table, which is populated from the original table, then swapped in its space.

    For disclosure, I authored gh-ost's code as part of the database infrastructure team at GitHub.

    Traditionally, online schema changes are considered to be "risky". Trigger based migrations add significant load onto the master server, and their cut-over phase is known to be a dangerous point. gh-ost was created at GitHub to address these concerns, and successfully eliminated concerns for operational risks: with gh-ost the load on the master is low, and well controlled, and the cut-over phase is known to cause no locking issues. gh-ost comes with different risks: it applies data changes programmatically, thus the issue of data integrity is of utmost importance. Another note of concern is data traffic: going out from MySQL into gh-ost and back into MySQL (as opposed to all-in MySQL in pt-online-schema-change).

    This way or the other, running an online schema change is typically a manual operation. A human being will schedule the migration, kick it running, monitor it, possibly cut-over. In a sharded environment, a developer's request to ALTER TABLE explodes to n different migrations, each needs to be scheduled, kicked, monitored & tracked.

    Sharded environments are obviously common for vitess users and so these users feel the pain more than others.

    Schema migration cycle & steps

    Schema management is a process that begins with the user designing a schema change, and ends with the schema being applied in production. This is a breakdown of schema management steps as I know them:

    1. Design code
    2. Publish changes (pull request)
    3. Review
    4. Formalize migration command (the specific ALTER TABLE or pt-online-schema-change or gh-ost command)
    5. Locate: where in production should this migration run?
    6. Schedule
    7. Execute
    8. Audit/monitor
    9. Cut-over/complete
    10. Cleanup
    11. Notify user
    12. Deploy & merge

    What we propose to address

    Vitess's architecture uniquely positions it to be able to automate away much of the process. Specifically:

    • Formalize migration command: turning an ALTER TABLE statement into a gh-ost invocation is super useful if done by vitess, since vitess can not only validate schema/params, but also can provide credentials, identify a throttle-control replica, can instruct gh-ost on how to communicate progress via hooks, etc.
    • Locate: given schema/table, vitess just knows where the table is located. It knows if the schema is sharded. It knows who the shards are, who the shards masters are. It knows where to run gh-ost. Last, vitess can tell us which replicas we can use for throttling.
    • Schedule: vitess is again in a unique position to schedule migrations. The fact someone asks for a migration to run does not mean the migration should start right away. For example, a shard may already be running an earlier migration. Running two migrations at a time is less than ideal, and it's best to wait out the first migration before beginning the second. A scheduling mechanism is both useful to running the migrations in optimal order/sequence, as well as providing feedback to the user ("your migration is on hold because this and that", or "your migration is 2nd in queue to run")
    • Execute: vttablet is the ideal entity to run a migration; can read instructions from topo server and can write progress to topo server. vitess is aware of possible master failovers and can request a re-execute is a migration is so interrupted mid process.
    • Audit/monitor: vtctld API can offer endpoints to track status of a migration (e.g. "in progress on -80, in queue on 80-"). It may offer progress pct and ETA.
    • cut-over/complete: in my experience with gh-ost, the cut-over phase is safe to automate away.
    • cleanup: the old table needs to be dropped; vttablet is in an excellent position to automate that away.

    What this PR does, and what we expect to achieve

    The guideline for this PR is: zero added dependencies; everything must be automatically and implicitly available via a normal vitess installation.

    A breakdown:

    User facing

    This PR enables the user to run an online schema migration (aka online DDL) via:

    • vtgate: the user connects to vitess with their standard MySQL client, and issues a ALTER WITH 'gh-ost' TABLE ... statement. Notice this isn't a valid MySQL syntax -- it's a hint for vitess that we want to run this migration online. vitess still supports synchronous, "normal" ALTER TABLE statements, which IMO should be discouraged.
    • vtctl: the user runs vtctl ApplySchema -sql "alter with _gh-ost' table ...".

    The response, in both cases, is a migration ID, or a job ID, if you will. Consider the following examples.

    via vtgate:

    
    mysql> create table example(id int auto_increment primary key, name tinytext);
    
    mysql> show create table example \G
    
    CREATE TABLE `example` (
      `id` int NOT NULL AUTO_INCREMENT,
      `name` tinytext,
      PRIMARY KEY (`id`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    
    mysql> alter with 'gh-ost' table example modify id bigint not null, add column status int, add key status_dx(status);
    +--------------------------------------+
    | uuid                                 |
    +--------------------------------------+
    | 211febfa-da2d-11ea-b490-f875a4d24e90 |
    +--------------------------------------+
    
    -- <wait...>
    
    mysql> show create table example \G
    
    CREATE TABLE `example` (
      `id` bigint NOT NULL,
      `name` tinytext,
      `status` int DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `status_dx` (`status`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    

    via vtctl:

    $ mysql -e "show create table example\G"
    
    CREATE TABLE `example` (
      `id` bigint NOT NULL,
      `name` tinytext,
      `status` int DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `status_dx` (`status`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    
    
    $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global \
        ApplySchema -sql "alter with 'gh-ost'  table example modify id bigint unsigned not null" commerce
    8ec347e1-da2e-11ea-892d-f875a4d24e90
    
    
    $ mysql -e "show create table example\G"
    
    CREATE TABLE `example` (
      `id` bigint unsigned NOT NULL,
      `name` tinytext,
      `status` int DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `status_dx` (`status`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    

    In both cases, a UUID is returned, which can be used for tracking (WIP) the progress of the migration across shards.

    Parser

    Vitess' parser now accepts ALTER WITH 'gh-ost' TABLE and ALTER WITH 'pt-osc' TABLE syntax. We're still to determine if this is the exact syntax we want to go with.

    Topo

    Whether submitted by vtgate or vtctl, we don't immediately run the migration. As mentioned before, we may wish to postpone the migration. Perhaps the relevant servers are already running a migration.

    Instead, we write the migration request into global topo, e.g.:

    • key: /vitess/global/schema-migration/requests/90c5afd4-da38-11ea-a3ff-f875a4d24e90
    • content:
    {"keyspace":"commerce","table":"example","sql":"alter table example modify id bigint not null","uuid":"90c5afd4-da38-11ea-a3ff-f875a4d24e90","online":true,"time_created":1596701930662801294,"status":"requested"}
    

    Once we create the request in topo, we immediately return the generated UUID/migration ID (90c5afd4-da38-11ea-a3ff-f875a4d24e90 in the above example) to the user.

    vtctld

    vtctld gets a conceptual "upgrade" with this PR. It is no longer a reactive service. vtctld now actively monitors new schema-migration/requests in topo.

    ~~When it sees such a request, it evaluates what are the relevant n shards.~~

    ~~With current implementaiton, it writes n "job" entries, one per shard. e.g.~~

    • /vitess/global/schema-migration/jobs/commerce/-80/ce45b84a-da2d-11ea-b490-f875a4d24e90 and /vitess/global/schema-migration/jobs/commerce/80-/ce45b84a-da2d-11ea-b490-f875a4d24e90 for a keyspace with two shards; or just
    • /vitess/global/schema-migration/jobs/commerce/0/1dd17132-da23-11ea-a3d2-f875a4d24e90 for a keyspace with one shard.

    DONE: WIP: we will investigate use of new VExec to actually distribute the jobs to vttablet.

    what vtctld does now, is, once it sees a migration request, it pushes a VExec request for that migration. If the VExec request succeeds, that means all shards have been notified, and vtctld can stow away the migration request (work is complete as far as vtctld is concerned). If VExec returns with an error, that means at least one shard did not get the request, and vtctld will keep retrying pushing this request.

    vttablet

    This is where most of the action takes place.

    vttablet runs a migration service which continuously probes for, schedules, and executes migrations.

    DONE: ~~With current implementation, tablets which have tablet_type=MASTER continuously probe for new entries. We look to replace this with VExec.~~

    migration requests are pushed via VExec; the request includes the INSERT IGNORE query that persists the migration in _vt.schema_migrations. The tablet no longer reads from, nor writes to, Global Topo.

    A new table is introduced: _vt.schema_migrations, which is how vttablet manages and tracks its own migrations.

    vttablet will only run a single migration at a time.

    vttablet will see if there's an unhandled migration requests. It will queue it.

    vttablet will make a migration ready if there's no running migration and no other migration is marked as ready.

    vttablet will run a ready migration. This is really the interesting part, with lots of goodies:

    • vttablet will evaluate the gh-ost ... command to run. It will obviously populate --alter=... --database=....
    • vttablet creates a temp directory where it generates a script to run gh-ost.
    • vttablet creates a hooks path and auto-generates hook files. The hooks will interact with vttablet
    • vttablet has an API endpoint by which the hooks can communicate gh-ost's status (started/running/success/failure) with vttablet.
    • vttablet provides gh-ost with --hooks-hint which is the migration's UUID.
    • vttablet automatically generates a gh-ost user on the MySQL server, with a random password. The password is never persisted and does not appear on ps. It is written to, and loaded from, an environment variable.
    • vttablet grants the proper privileges on the newly created account
    • vttablet will destroy the account once migration completes.
    • vitess repo includes a gh-ost binary. We require gh-ost from openark/gh-ost as opposed to github/gh-ost because we've had to make some special adjustments to gh-ost s oas to support this flow. I do not have direct ownership to github/gh-ost and cannot enforce those changes upstream, though I have made the contribution requestss upstream.
    • make build automatically appends gh-ost binary, compressed, to vttablet binary, via Ricebox.
    • vttablet, upon startup, auto extracts gh-ost binary into /tmp/vt-gh-ost. Please note that the user does not need to install gh-ost.
    • WIP: vttablet to report back the job as complete/failed. We look to use VExec. TBD.

    Tracking breakdown

    • [x] New OnlineDDL struct, defines a migration request and its status
    • [x] Parser supports ALTER WITH 'gh-ost' TABLE and ALTER WITH 'pt-osc' TABLE syntax
    • [x] builder and analyzer to create an Online DDL plan (write to topo)
    • [x] vtctl to skip "big changes" check when -online_schema_change is given
    • [x] tablet_executor to submit an online DDL request to topo as opposed to running it on tablets
    • [x] vtctld runs a daemon to monitor for, and review migration requests
    • [x] vtctld evaluates which shards are affected
    • [x] _vt.schema_migrations backend table to support migration automation (on each shard))
    • [x] vttablet validates MySQL connection and variables
    • [x] vttablet creates migration command
    • [x] vttablet creates hooks
    • [x] vttablet provides HTTP API for hooks to report their status back
    • [x] vttablet creates gh-ost user with random password
    • [x] vttablet destroys gh-ost user upon completion
    • [x] gh-ost embedded in vttablet binary and auto-extracted by vttablet
    • [x] vttablet runs a dry-run execution
    • [x] vttablet runs a --execute (actual) execution
    • [x] vttablet supports a Cancel request (not used yet) to abort migration
    • [x] vttablet as a state machine to work throught the migration steps
    • [x] counters for gh-ost migration requests, suceessful and failed migrations
    • [x] use of VExec to apply migrations onto tablets
    • [x] use of VExec to control migrations (abort, retry)
    • [ ] consider flow for retries
    • [ ] identify a reparent operation that runs during a migration, probabaly auto-restart the migration
    • [ ] vttablet to heuristically check for available disk space
    • [x] tracking, auditing of all migrations
    • [x] getting gh-ost logs if necessary
    • [x] what's the best way to suggest we want an online migration? Does current ALTER WITH 'gh-ost' TABLE... and ALTER WITH 'pt-osc' TABLE syntax make sense? Other?
    • [ ] For first iteration, migrations and Reshard operations should be mutually exclusive. Can't run both at the same time. Next iterations will remove this constraint.
    • [x] ~~throttle by replica~~
    • [ ] ~~wait for replica to catch up with new credentials before starting the migration~~
    • [x] Use vttablet throttler
    • [x] pt-online-schema-change bundled inside vttablet binary
    • [x] support pt-online-schema-change
    • [ ] ~~define foreign key flags for pt-online-schema-change execution~~ - user can define as runtime flags
    • [x] clenaup online-ddl directory after success
    • [ ] control throttling
    • [x] control termination (panic abort)
    • [x] control termination (panic abort) even after vttablet itself crashes
    • [x] pt-online-schema-change passwords are in cleartext. Can we avoid that?
    • [x] vtctl ApplySchema use same WITH 'gh-ost' and WITH 'pt-osc' query hints as in vtgate.
    • [x] support override of gh-ost and pt-online-schema-change paths
    • [x] cleanup pt-osc triggers after migration failure
    • [x] forcibly remove pt-osc triggers on migration cancellation (overlaps with previous bullet, but has stronger guarantee)
    • [x] cleanup pt-osc triggers from stale/zombie pt-osc migration
    • [x] vtctl OnlineDDL command for simple visibility and manipulation. See https://github.com/vitessio/vitess/pull/6547#issuecomment-681879259
    • [x] end to end tests
    • [x] populate artifacts column, suggesting which tables need to be cleaned up after migration

    Quite likely more entries to be added.

    Further reading, resources, acknowledgements

    We're obviously using gh-ost. I use my own openark/gh-ost since I have no ownership of the original https://github.com/github/gh-ost. gh-ost was/is developed by GitHub 2016-2020.

    pt-online-schema-change is part of the popular Percona Toolkit

    The schema migratoin scheduling and tracking work is based on my previous work at GitHub. The implementation in this PR is new and rewritten, but based on concepts that have matured on my work on skeefree. Consider these resources:

    Also:

    • An early presentation on gh-ost

    Initial incarnation of this PR: https://github.com/planetscale/vitess/pull/67; some useful comments on that PR.

    Call for feedback

    We're looking for community's feedback on the above suggestions/flow. Thank you for taking the time to read and respond!

    Type: Enhancement 
    opened by shlomi-noach 65
  • Adding TLS support to the Java client and JDBC driver

    Adding TLS support to the Java client and JDBC driver

    re: Issue #2209

    TLS connections are already supported by VTGate, gRPC, and all of the non-Java clients. For Java to reach parity, support is needed in the grpc-client, client, and jdbc modules under src/github.com/youtube/vitess/java.

    This PR is a work in progress... visible for early review and vetting, but not yet ready to merge.

    At this point, the VitessJDBCUrl class parses an optional parameter, which indicates that a connection should use SSL. The parameter follows standard MySQL JDBC conventions.

    Pending commits will:

    1. Update GrpcClientFactory, to expose a method for creating a TLS-enabled RpcClient in addition to the current plaintext one.
    2. Update VitessVTGateManager, to use this new method when the JDBC URL calls for SSL.
    3. Write unit test coverage for the above, using a vtgateclienttest mock in a style similar to the src/github.com/youtube/vitess/test/encrypted_transport.py script.
    4. End-to-end testing. Update examples/local/vtgate-up.sh, to use TLS with a self-signed cert when an optional parameter is passed. Also update the examples/client_jdbc.sh test script, to likewise use a TLS connection when an optional parameter is passed.

    Any feedback about this approach is definitely welcomed!


    This change is Reviewable

    opened by steve-perkins 63
  • Single shard targeting in V3

    Single shard targeting in V3

    There is no way to target a single shard in the V3 APIs.

    We need to target single shards for two reasons:

    1. We want to shard some of our processing in that we have jobs that process an entire single shard at the time.
    2. There's a very large number of scatter/gather functionality that is currently not supported (e.g. limit on joins, order by and so on). Some we can work around on the query level but some we have to build our own scatter/gather logic.

    Item 2 could eventually be addressed by extending scatter/gather in V3 but item 1 can never be addressed by V3.

    opened by tirsen 31
  • easier build

    easier build

    I worked on this because I finished a brew formula to install Vitess on macOS with one command: brew install vitess. Also, there was some conversation on the Slack channel to improve the build process and make it easier.

    The main change in here is: we will push the vendor libs, instead of getting them during the bootstrap.sh script run. Advantages of this approach:

    • We ensure the version of our libs, not just by the vendor.json file, but they actually will follow the history of this repo.
    • Makes the build process a lot easier. With this, I can build Vitess without needing to run boostrap.sh. This is useful for folks that don't need to change the code but just want to build it.
    • Homebrew build process doesn't allow us to do go get during it, as that is insecure. But, as we vendor our dependencies we don't need to go get during build time anymore.

    Drawbacks:

    • when changing the vendor.json file, we need to run go vendor sync or the bootstrap script again, to upload the new version of the lib.
    • This repository will get bigger in git size. (the local size will still be the same, though)

    I think the advantages outweighs the disadvantages, as the ratio of local builds vs changing vendor.json file is bigger.

    At, least at GitHub, thats what we do, we vendored and upload the lib sources to git. Also, open source projects such as Consul do the same.

    Thoughts? cc @sougou @vmg

    opened by arthurnn 30
  • helm: Allow non-leader Orchestrator instances to accept requests

    helm: Allow non-leader Orchestrator instances to accept requests

    Orchestrator 3.0.7 added a proxy that forwards master only requests, so we don’t have to workaround that by having perpetually unready pods via the /api/leader-check endpoint

    cc @shlomi-noach @enisoc

    opened by derekperkins 30
  • Adds table name to the field proto

    Adds table name to the field proto

    https://dev.mysql.com/doc/refman/5.7/en/c-api-data-structures.html

    The table and table_length fields are available in the C api but are not passed through to clients for use. One of our core java client libraries depends on http://docs.oracle.com/javase/7/docs/api/java/sql/ResultSetMetaData.html#getTableName(int), which in the existing Vitess Driver was hardcoded to return null.

    This PR adds table to the Field proto, populates it in the mysql.Fields() function, and allows access in the Java client. All tests updated to account for the new field, and succeeding.

    I've also tested this in our deployment. However, I've not been able to get the full make test to work in my local dev yet, this was tested with make site_test.

    This is my first foray into both go and vitess, so feedback welcome.

    opened by bbeaudreault 28
  • Preserve case of column names

    Preserve case of column names

    When Vitess receives a query, the tokenizer lowercases all column names so they can be string-compared safely throughout the system. This means all column names returned in the eventual query result are lowercase as well. This causes a problem in PHP when users expect to look up columns by name with the same case they requested in the original query:

    $result = query('select Column from table');
    $row = $result->fetch();
    echo $row['Column']; // array key doesn't exist
    echo $row['column']; // works
    

    In Java (JDBC) they address this by specifying that column name lookups are case-insensitive. We recently fixed our Java client to conform to this (https://github.com/youtube/vitess/pull/1572).

    We could do something similar in PHP by implementing ArrayAccess or extending ArrayObject. Each of those has its pros and cons, but either way, the object we return would not be a true array, so things like array_keys() wouldn't work.

    @tslater @Rastusik Can you weigh in on whether this makes sense from a PHP app writer's perspective?

    @sougou

    Type: Feature Request 
    opened by enisoc 27
  • RFC: Deprecate MariaDB Support in 2022 and Remove in 2023

    RFC: Deprecate MariaDB Support in 2022 and Remove in 2023

    Proposal

    Deprecate official MariaDB support in Vitess 14.0 (~June 2022) and remove it entirely in 16.0 (~Feb 2023). This gives Vitess users approximately 2 years where they can continue to use Vitess with MariaDB and offers adequate time to transition to MySQL 8 (MySQL 5.7 is currently scheduled for EOL in 2023).

    Reasoning

    Note: MySQL and Percona Server for MySQL are considered fully equivalent in Vitess.

    The reasons for proposing this are as follows:

    • Vitess only officially supports MariaDB 10.0-10.3 today
    • MariaDB is a hard fork of MySQL and 10.X is a very different database than MySQL 8.X (which is quickly becoming the default with Vitess)
    • Though there was some outreach from the MariaDB Foundation last year to get Vitess working with more recent versions of MariaDB, that effort seems to have been abandoned
    • MariaDB's GTID implementation — which is NOT compatible with MySQL which uses auto generated UUIDs for the unique host identifier — uses domain_id and server_id together as the "unique" host identifier while every MariaDB instance defaults to 0-1 for its unique host identifier which is in NO way unique. MariaDB's GTID implementation also do not use sets (MySQL uses GTID_EXECUTED and GTID_PURGED sets), but only the last seen sequence value from the given host identifier (so e.g. 0-1:20000 vs 15b57a66-e10d-11eb-a4de-7499a366173e:1-20000 in MySQL) — so you have no way to know if two servers have executed the same full set of GTIDs or not and you cannot easily detect drift and ensure consistency and correctness. When you combine these two things, this is a big problem for Vitess, which relies on fairly complex replication topologies. This makes MariaDB unsafe to use with Vitess VReplication, and you will run into problems (surfaced duplicate GTIDs are the least of your problems, the worse issue is undetected drift and inconsistencies within a shard). This is hardly the only problem with MariaDB usage in Vitess, but I assert that this alone makes MariaDB an unsafe and unsuitable choice.

    And most importantly:

    • Vitess is not well tested on MariaDB today within the project nor the user base
      • There are no known Vitess users with MariaDB in production (please let us know if you do!)

    Given all of this, it seems reasonable to discontinue spending the limited human and capital resources available to the Vitess project — which would otherwise go toward fixing bugs and adding features in Vitess — in trying to maintain official MariaDB support (at least for 10.4+) going forward.

    :warning: While we would not officially support MariaDB as a database within Vitess, we would still want to maintain a clear migration path into Vitess for existing MariaDB installations.

    Feedback

    Please note that we want to do what's best for the Vitess project and its community, so your input is crucial! This is a proposal, it is not an announcement of a decision that's been made. If you have any questions, concerns, or other feedback please let us know!

    Type: RFC 
    opened by mattlord 26
  • grpc build problem

    grpc build problem

    While executing ./bootstrap.sh on OSX I get the following error:

    [HOSTCXX] Compiling src/compiler/csharp_generator.cc
    src/compiler/csharp_generator.cc:47:43: error: no member named 'GetReflectionClassName' in namespace 'google::protobuf::compiler::csharp'
    using google::protobuf::compiler::csharp::GetReflectionClassName;
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
    src/compiler/csharp_generator.cc:237:26: error: use of undeclared identifier 'GetReflectionClassName'
                 "umbrella", GetReflectionClassName(service->file()), "index",
                             ^
    2 errors generated.
    make: *** [/Users/yfinkelstein/work/vitess_workspace/dist/grpc/grpc/objs/opt/src/compiler/csharp_generator.o] Error 1
    ERROR: gRPC build failed
    

    I'm following build setup instructions line by line as described in your "How to build".

    opened by yfinkelstein 26
  • Added transaction-throttler and connect it to tabletserver.

    Added transaction-throttler and connect it to tabletserver.

    It uses vitess/go/vt/throttler with healthchecks to throttle transactions based on replication lag.

    @sougou @michael-berlin please take a look. Please pay extra attention to my integration code in tabletserver.go. I wanted the throttler to be up only when the tablet is a master and I'm unfamiliar with the state-transitioning code in the tablet. Please make sure I didn't make any mistakes.

    Thanks!

    Erez.

    opened by erzel 25
  • Add more robust go version handling

    Add more robust go version handling

    Description

    Go 1.19 was just released and as reported in https://github.com/vitessio/vitess/issues/11000 it caused builds to fail when installed:

    ERROR: Go version reported: go version go1.19 linux/amd64. Version 1.18.4+ required. See https://vitess.io/contributing/build-from-source for install instructions.
    make: *** [Makefile:85: build] Error 1
    

    To demonstrate the new handling added in this PR:

    $ go version
    go version go1.19 darwin/arm64
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min || fail "No good"'
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min 1 || fail "No good"'
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min 2 || fail "No good"'
    ERROR: No good
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min 1.18.9999-foo || fail "No good"'
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min 1.19 || fail "No good"'
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min 1.19.0 || fail "No good"'
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min 1.19.1 || fail "No good"'
    ERROR: No good
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min 1.19.55-dev || fail "No good"'
    ERROR: No good
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min 1.20 || fail "No good"'
    ERROR: No good
    
    $ bash -c 'source ./tools/shell_functions.inc && goversion_min 1.20.2 || fail "No good"'
    ERROR: No good
    

    Related Issue(s)

    • Fixes: https://github.com/vitessio/vitess/issues/11000

    Checklist

    • [x] "Backport me!" label has been added if this change should be backported
    • [x] Tests are not required
    • [x] Documentation is not required
    Component: Build/CI Type: CI/Build Backport to: release-14.0 Backport to: release-13.0 
    opened by mattlord 1
  • Cannot build with go 1.19

    Cannot build with go 1.19

    Overview of the Issue

    Error from make build

    ERROR: Go version reported: go version go1.19 linux/amd64. Version 1.18.4+ required. See https://vitess.io/contributing/build-from-source for install instructions.
    make: *** [Makefile:85: build] Error 1
    

    If you remove the goversion_min check from build.env it fails later with

    compile: version "go1.18.5" does not match go tool version "go1.19"
    

    Reproduction Steps

    Install go 1.19 and try to build from a vitess local checkout

    Binary Version

    main
    

    Operating System and Environment details

    MacOS M1
    

    Log Fragments

    ERROR: Go version reported: go version go1.19 linux/amd64. Version 1.18.4+ required. See https://vitess.io/contributing/build-from-source for install instructions.
    make: *** [Makefile:85: build] Error 1
    
    Type: Bug Component: Build/CI 
    opened by deepthi 1
  • [cli] [tabletconn] Migrate `go/vt/vttablet/tabletconn` to `pflag`

    [cli] [tabletconn] Migrate `go/vt/vttablet/tabletconn` to `pflag`

    Description

    Still some TODOs which I will amend this commit.

    Here's my analysis:

    vitess.io/vitess/go/cmd/rulesctl => nope
        vitess.io/vitess/go/cmd/rulesctl/cmd
        vitess.io/vitess/go/cmd/rulesctl/common
    vitess.io/vitess/go/cmd/vtadmin => nope
    vitess.io/vitess/go/cmd/vtbench => yep
    vitess.io/vitess/go/cmd/vtcombo => yep
    vitess.io/vitess/go/cmd/vtctl => yep
    vitess.io/vitess/go/cmd/vtctld => yep
    vitess.io/vitess/go/cmd/vtctldclient => yes (only whvten using local shim, which _actually_ happens via `go/cmd/vtctl`)
        vitess.io/vitess/go/cmd/vtctldclient/command
        vitess.io/vitess/go/cmd/vtctldclient/docgen
    vitess.io/vitess/go/cmd/vtexplain => ?
    vitess.io/vitess/go/cmd/vtgate => yep
    vitess.io/vitess/go/cmd/vtgateclienttest => ?
    vitess.io/vitess/go/cmd/vttablet => yep (for streaming from other tablets)
    vitess.io/vitess/go/cmd/zk => nope (imports `vtctl` for `DecodeContent` only)
    

    Related Issue(s)

    Closes #10978.

    Checklist

    • [ ] "Backport me!" label has been added if this change should be backported
    • [x] Tests were added or are not required
    • [x] Documentation was added or is not required

    Deployment Notes

    Type: Internal Cleanup Type: Enhancement Component: Cluster management Component: Query Serving Do Not Merge Component: CLI 
    opened by ajm188 1
  • evalengine: support mismatched numerical types

    evalengine: support mismatched numerical types

    Description

    Found this while working on some internal stuff. The evalengine does not properly handle mismatched numeric types because we were not properly normalizing. E.g. an UINT8(0) would fail to compare equally to an UINT64(0), even though an INT8(0) would actually compare equally to a UINT64(0). This PR fixes this issue by upcasting all comparison types before performing a numeric comparison.

    It probably requires a backport.

    cc @systay @deepthi

    Related Issue(s)

    Checklist

    • [x] "Backport me!" label has been added if this change should be backported
    • [x] Tests were added or are not required
    • [x] Documentation was added or is not required

    Deployment Notes

    Type: Bug Component: Evalengine Backport to: release-14.0 Backport to: release-13.0 
    opened by vmg 4
  • `vtexplain` fails for vindex lookup queries with duplicate / equivalent values.

    `vtexplain` fails for vindex lookup queries with duplicate / equivalent values.

    Description

    If a query contains an IN condition with multiple equivalent values for lookup_unique vindexes, vtexplain fails with:

    Lookup.Map: unexpected multiple results from vindex <vindex>
    

    This is due to vtexplain "simulating" that each value of the tuple in a lookup vindex query returns a result row, but when values are equivalent (e.g. 1 and "1"), only a single row should be returned.

    I tried to implement this by determining the type of the column being matched against, converting each value of the tuple to that type, and then making sure vtexplain only considers non-duplicate values.

    Related Issue(s)

    Checklist

    • [ ] "Backport me!" label has been added if this change should be backported
    • [ ] Tests were added or are not required
    • [ ] Documentation was added or is not required

    Deployment Notes

    opened by arthurschreiber 1
  • Switch flag definitions to be on pflag instead of flag in `package go/cmd/vtcombo`

    Switch flag definitions to be on pflag instead of flag in `package go/cmd/vtcombo`

    Part of https://github.com/vitessio/vitess/issues/10697.

    Current flags:

    $ git grep -E "\bflag\.[A-Z]" -- go/cmd/vtcombo/*.go
    go/cmd/vtcombo/main.go: schemaDir          = flag.String("schema_dir", "", "Schema base directory. Should contain one directory per keyspace, with a vschema.json file if necessary.")
    go/cmd/vtcombo/main.go: startMysql         = flag.Bool("start_mysql", false, "Should vtcombo also start mysql")
    go/cmd/vtcombo/main.go: mysqlPort          = flag.Int("mysql_port", 3306, "mysql port")
    go/cmd/vtcombo/main.go: externalTopoServer = flag.Bool("external_topo_server", false, "Should vtcombo use an external topology server instead of starting its own in-memory topology server. "+
    go/cmd/vtcombo/main.go: plannerVersion           = flag.String("planner-version", "", "Sets the default planner to use when the session has not changed it. Valid values are: V3, Gen4, Gen4Greedy and Gen4Fallback. Gen4Fallback tries the gen4 planner and falls back to the V3 planner if the gen4 fails.")
    go/cmd/vtcombo/main.go: plannerVersionDeprecated = flag.String("planner_version", "", "Deprecated flag. Use planner-version instead")
    go/cmd/vtcombo/main.go: flag.Var(vttest.TextTopoData(&tpb), "proto_topo", "vttest proto definition of the topology, encoded in compact text format. See vttest.proto for more information.")
    go/cmd/vtcombo/main.go: flag.Var(vttest.JSONTopoData(&tpb), "json_topo", "vttest proto definition of the topology, encoded in json format. See vttest.proto for more information.")
    go/cmd/vtcombo/main.go: flag.Set("cells_to_watch", strings.Join(tpb.Cells, ","))
    go/cmd/vtcombo/main.go: flag.Set("cell", tpb.Cells[0])
    go/cmd/vtcombo/main.go: flag.Set("enable_realtime_stats", "true")
    go/cmd/vtcombo/main.go: if flag.Lookup("log_dir") == nil {
    go/cmd/vtcombo/main.go:         flag.Set("log_dir", "$VTDATAROOT/tmp")
    
    Type: Internal Cleanup Type: Enhancement Component: VTCombo Component: CLI 
    opened by ajm188 0
Releases(v12.0.5)
  • v12.0.5(Jul 27, 2022)

    Release of Vitess v12.0.5

    Major Changes

    Upgrade to go1.17.12

    Vitess v12.0.5 now runs on go1.17.12. The patch release of Go, go1.17.12, is the reason for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.17.12 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the runtime, and the runtime/metrics package. See the Go 1.17.12 milestone on our issue tracker for details.


    The entire changelog for this release can be found here.

    The release includes 7 commits (excluding merges)

    Thanks to all our contributors: @deepthi, @frouioui

    Source code(tar.gz)
    Source code(zip)
  • v14.0.1(Jul 26, 2022)

    Release of Vitess v14.0.1

    Known Issues

    Major Changes

    Upgrade to go1.18.4

    Vitess v14.0.1 now runs on go1.18.4. The patch release of Go, go1.18.4, was one of main motivations for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.18.4 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the linker, the runtime, and the runtime/metrics package. See the Go 1.18.4 milestone on our issue tracker for details.


    The entire changelog for this release can be found here.

    The release includes 25 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @deepthi, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @shlomi-noach, @vitess-bot[bot]

    Source code(tar.gz)
    Source code(zip)
    vitess-14.0.1-4c118cb.tar.gz(488.83 MB)
    vitess-14.0.1-4c118cb.x86_64.rpm(487.08 MB)
    vitess_14.0.1-4c118cb_amd64.deb(488.85 MB)
  • v13.0.2(Jul 26, 2022)

    Release of Vitess v13.0.2

    Major Changes

    Upgrade to go1.17.12

    Vitess v13.0.2 now runs on go1.17.12. The patch release of Go, go1.17.12, was one of main motivations for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.17.12 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the runtime, and the runtime/metrics package. See the Go 1.17.12 milestone on our issue tracker for details.


    The entire changelog for this release can be found here.

    The release includes 28 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @aquarapid, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @systay, @vitess-bot[bot], @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-13.0.2-60dc8b9.tar.gz(463.35 MB)
    vitess-13.0.2-60dc8b9.x86_64.rpm(461.69 MB)
    vitess_13.0.2-60dc8b9_amd64.deb(463.36 MB)
  • v14.0.0(Jun 28, 2022)

    Release of Vitess v14.0.0

    Summary

    Known Issues

    Major Changes

    Gen4 is now the default planner

    The new planner has been in the works since end of 2020, and it's finally grown enough to be able to become the default planner for Vitess. This means that many more queries are supported on sharded keyspaces, and old queries might get planned better than before. You can always roll back to the earlier planner, either by providing the flag --planner-version=V3 to vtgate, or by adding a comment to individual queries, like so:

    select /*vt+ PLANNER=V3 */ name, count(*) from users
    

    Schema Tracking is no longer experimental

    Schema tracking is now GA. It is enabled by default. Schema tracking allows the Gen4 planner to plan many more queries. This can be turned off if necessary by changing the relevant flags on VTGate and VTTablet.

    New query support

    Support for aggregation across shards

    Vitess can now plan and execute most aggregation queries across multiple shards and/or keyspaces.

    INSERT from SELECT

    Support has been added for inserting new data from SELECT queries. Now you can insert data from a query into a table using a query like:

    insert into tbl (col) select id from users 
    

    UPDATE from SELECT

    Similarly, we have added support for UPDATE with scalar sub-queries. This allows for queries where the updated value is fetched using a subquery, such as this example:

    update tbl set foo = (select count(*) from otherTbl)
    

    Command-line syntax deprecations

    Vitess has begun a transition to a new library for CLI flag parsing. In order to facilitate a smooth transition, certain syntaxes that will not be supported in the future now issue deprecation warnings when used.

    The messages you will likely see, along with explanations and migrations, are:

    "Use of single-dash long flags is deprecated"

    Single-dash usage will be only possible for short flags (e.g. -v is okay, but -verbose is not).

    To migrate, update your CLI scripts from:

    $ vttablet -tablet_alias zone1-100 -init_keyspace mykeyspace ... # old way
    

    To:

    $ vttablet --tablet_alias zone1-100 --init_keyspace mykeyspace ... # new way
    

    "Detected a dashed argument after a position argument."

    As the full deprecation text goes on to (attempt to) explain, mixing flags and positional arguments will change in a future version that will break scripts.

    Currently, when invoking a binary like:

    $ vtctl --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
    

    Everything after the AddCellInfo is treated by package flag as a positional argument, and we then use a sub FlagSet to parse flags specific to the subcommand. So, at the top-level, flag.Args() returns ["AddCellInfo", "--root", "/vitess/global"].

    The library we are transitioning to is more flexible, allowing flags and positional arguments to be interwoven on the command-line. For the above example, this means that we would attempt to parse --root as a top-level flag for the VTCtl binary. This will cause the program to exit on error, because that flag is only defined on the AddCellInfo subcommand.

    In order to transition, a standalone double-dash (literally, --) will cause the new flag library to treat everything following that as a positional argument, and also works with the current flag parsing code we use.

    So, to transition the above example without breakage, update the command to:

    $ vtctl --topo_implementation etcd2 AddCellInfo -- --root "/vitess/global"
    $ # the following will also work
    $ vtctl --topo_implementation etcd2 -- AddCellInfo --root "/vitess/global"
    $ # the following will NOT work, because --topo_implementation is a top-level flag, not a sub-command flag
    $ vtctl -- --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
    

    New command line flags and behavior

    vttablet --heartbeat_on_demand_duration

    --heartbeat_on_demand_duration joins the already existing heartbeat flags --heartbeat_enable and --heartbeat_interval and adds new behavior to heartbeat writes.

    --heartbeat_on_demand_duration takes a duration value, such as 5s.

    The default value for --heartbeat_on_demand_duration is zero, which means the flag is not set and there is no change in behavior.

    When --heartbeat_on_demand_duration has a positive value, then heartbeats are only injected on demand, based on internal requests. For example, when --heartbeat_on_demand_duration=5s, the tablet starts without injecting heartbeats. An internal module, like the lag throttler, may request the heartbeat writer for heartbeats. Starting at that point in time, and for the duration (a lease) of 5s in our example, the tablet will write heartbeats. If no other requests come in during that time, the tablet then ceases to write heartbeats. If more requests for heartbeats come in, the tablet extends the lease for the next 5s following each request. It stops writing heartbeats 5s after the last request is received.

    The heartbeats are generated according to --heartbeat_interval.

    Deprecation of --online_ddl_check_interval

    The flag --online_ddl_check_interval is deprecated and will be removed in v15. It has been unused in v13.

    Removal of --gateway_implementation

    In previous releases, the discoverygateway was deprecated. In Vitess 14 it is now entirely removed, along with the VTGate flag that allowed us to choose a gateway.

    Deprecation of --planner_version

    The flag --planner_version is deprecated and will be removed in v15. Some binaries used --planner_version, and some used --planner-version. This has been made consistent - all binaries that allow you to configure the planner now take --planner-version. All uses of the underscore form have been deprecated and will be removed in v15.

    Online DDL changes

    Online DDL is generally available

    Online DDL is no longer experimental (with the exception of pt-osc strategy). Specifically:

    • Managed schema changes, the scheduler, the backing tables
    • Supporting SQL syntax
    • vitess strategy (online DDL via VReplication)
    • gh-ost strategy (online DDL via 3rd party gh-ost)
    • Recoverable migrations
    • Revertible migrations
    • Declarative migrations
    • Postponed migrations
    • And all other functionality

    Are all considered production-ready.

    pt-osc strategy (online DDL via 3rd party pt-online-schema-change) remains experimental.

    ddl_strategy: 'vitess'

    ddl_strategy now takes the value of vitess to indicate VReplication-based migrations. It is a synonym to online and uses the exact same functionality. The online term will be phased out in the future and vitess will remain the term of preference.

    Example:

    vtctlclient ApplySchema -skip_preflight -ddl_strategy='vitess' -sql "alter table my_table add column my_val int not null default 0" commerce
    

    --singleton-context and REVERT migrations

    It is now possible to submit a migration with --singleton-context strategy flag, while there's a pending (queued or running) REVERT migration that does not have a --singleton-context flag.

    Support for CHECK constraints

    Online DDL operations are more aware of CHECK constraints, and properly handle the limitation where a CHECK's name has to be unique in the schema. As opposed to letting MySQL choose arbitrary names for shadow table's CHECK constraints, Online DDL now generates unique yet deterministic names, such that all shards converge onto the same names.

    Online DDL attempts to preserve the original check's name as a suffix to the generated name, where possible (names are limited to 64 characters).

    Behavior changes

    • vtctl ApplySchema --uuid_list='...' now rejects a migration if an existing migration has the same UUID but with different migration_context.

    Table lifecycle

    Views

    Table lifecycle now supports views. It does not purge rows from views, and does not keep views in EVAC state (they are immediately transitioned to DROP state).

    Fast drops

    On Mysql 8.0.23 or later, the states PURGE and EVAC are automatically skipped, thanks to 8.0.23 improvements to DROP TABLE speed of operation.

    Tablet throttler

    API changes

    Added /throttler/throttled-apps endpoint, which reports back all current throttling instructions. Note, this only reports explicit throttling requests (such as ones submitted by /throtler/throttle-app?app=...). It does not list incidental rejections based on throttle thresholds.

    API endpoint /throttler/throttle-app now accepts a ratio query argument, a floating point value in the range [0..1], where:

    • 0 means "do not throttle at all"
    • 1 means "always throttle"
    • Any number in between is allowed. For example, 0.3 means "throttle with 0.3 probability", i.e. for any given request there's a 30% chance that the request is denied. Overall we can expect about 30% of requests to be denied. Example: /throttler/throttle-app?app=vreplication&ratio=0.25.

    See new SQL syntax for controlling/viewing throttling, under New Syntax.

    New Syntax

    Control and view Online DDL throttling

    We introduce the following syntax to:

    • Start/stop throttling for all Online DDL migrations, in general
    • Start/stop throttling for a particular Online DDL migration
    • View throttler state
    ALTER VITESS_MIGRATION '<uuid>' THROTTLE [EXPIRE '<duration>'] [RATIO <ratio>];
    ALTER VITESS_MIGRATION THROTTLE ALL [EXPIRE '<duration>'] [RATIO <ratio>];
    ALTER VITESS_MIGRATION '<uuid>' UNTHROTTLE;
    ALTER VITESS_MIGRATION UNTHROTTLE ALL;
    SHOW VITESS_THROTTLED_APPS;
    

    The default duration is "infinite" (set as 100 years):

    • Allowed units are (s)ec, (m)in, (h)our

    The ratio is in the range [0..1]:

    • 1 means throttle everything - the app will not make any progress
    • 0 means no throttling at all
    • 0.8 means on 8 out of 10 checks the app makes, it gets refused

    The syntax SHOW VITESS_THROTTLED_APPS is a generic call to the throttler, and returns information about all throttled apps, not specific to migrations.

    The output of SHOW VITESS_MIGRATIONS ... now includes user_throttle_ratio.

    This column is updated "once in a while", while a migration is running. Normally this is once a minute, but can be more frequent. The migration reports back the throttling instruction set by the user while it was running. This column does not indicate any lag-based throttling that might take place based on the throttler configuration. It only reports the explicit throttling value set by the user.

    Heartbeat

    The throttler now checks in with the heartbeat writer to request heartbeats, any time it (the throttler) is asked for a check.

    When --heartbeat_on_demand_duration is not set, there is no change in behavior.

    When --heartbeat_on_demand_duration is set to a positive value, then the throttler ensures that the heartbeat writer generates heartbeats for at least the following duration. This also means at the first throttler check, it's possible that heartbeats are idle, and so the first check will fail. As heartbeats start running, followup checks will get a more accurate lag evaluation and will respond accordingly. In a sense, it's a "cold engine" scenario, where the engine takes time to start up, and then runs smoothly.

    VDiff2

    We introduced a new version of VDiff -- currently marked as EXPERIMENTAL -- that executes the VDiff on vttablets rather than in vtctld. While this is experimental we encourage you to try it out and provide feedback! This input will be invaluable as we improve the feature on the march toward a production-ready version. You can try it out by adding the --v2 flag to your VDiff command. Here's an example:

    $ vtctlclient --server=localhost:15999 VDiff -- --v2 customer.commerce2customer
    VDiff bf9dfc5f-e5e6-11ec-823d-0aa62e50dd24 scheduled on target shards, use show to view progress
    
    $ vtctlclient --server=localhost:15999 VDiff -- --v2 customer.commerce2customer show last
    
    VDiff Summary for customer.commerce2customer (4c664dc2-eba9-11ec-9ef7-920702940ee0)
    State:        completed
    RowsCompared: 196
    HasMismatch:  false
    StartedAt:    2022-06-26 22:44:29
    CompletedAt:  2022-06-26 22:44:31
    
    Use "--format=json" for more detailed output.
    
    $ vtctlclient --server=localhost:15999 VDiff -- --v2 --format=json customer.commerce2customer show last
    {
    	"Workflow": "commerce2customer",
    	"Keyspace": "customer",
    	"State": "completed",
    	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0",
    	"RowsCompared": 196,
    	"HasMismatch": false,
    	"Shards": "0",
    	"StartedAt": "2022-06-26 22:44:29",
    	"CompletedAt": "2022-06-26 22:44:31"
    }
    

    Even before it's marked as production-ready (feature complete and tested widely in 1+ releases), it should be safe to use and is likely to provide much better results for very large tables.

    For additional details please see the RFC, the README, and the VDiff2 documentation.

    Durability Policy

    Deprecation of durability_policy Flag

    The durability policy for a keyspace is now stored in the keyspace record in the topology server. The durability_policy flag used by VTCtl, VTCtld, and VTWorker binaries has been deprecated and will be removed in a future release.

    New and Augmented Commands

    The VTCtld command CreateKeyspace has been augmented to take in an additional argument --durability-policy which will allow users to set the desired durability policy for a keyspace at creation time.

    For existing keyspaces, a new command SetKeyspaceDurabilityPolicy has been added, which allows users to change the durability policy of an existing keyspace.

    If semi-sync is not being used then durability policy should be set to none for the keyspace. This is also the default option.

    If semi-sync is being used then durability policy should be set to semi_sync for the keyspace and --enable_semi_sync should be set on vttablets.

    VTOrc - Deprecation of Durability Configuration

    The Durability configuration is deprecated and removed from VTOrc. Instead VTOrc will find the durability policy of the keyspace from the topology server. This allows VTOrc to monitor and repair multiple keyspaces which have different durability policies in use.

    VTOrc will ignore keyspaces which have no durability policy specified in the keyspace record. This is to avoid clobbering an existing config from a previous release. So on upgrading to v14, users must run the command SetKeyspaceDurabilityPolicy specified above, to ensure that VTOrc continues to work as desired. The recommended upgrade path is to upgrade VTCtld, run SetKeyspaceDurabilityPolicy and then upgrade VTOrc.

    Advisory locking optimizations

    Work has gone into making the advisory locks (get_lock(), release_lock(), etc.) release reserved connections faster and in more situations than before.

    Pre-Legacy Resharding is now deprecated

    A long time ago, the sharding column and type were specified at the keyspace level. This syntax is now deprecated and will be removed in v15.


    The entire changelog for this release can be found here.

    The release includes 1101 commits (excluding merges)

    Thanks to all our contributors: @FancyFane, @GuptaManan100, @Juneezee, @K-Kumar-01, @Phanatic, @ajm188, @akenneth, @aquarapid, @arthurschreiber, @brendar, @cuishuang, @dasl-, @dbussink, @deepthi, @dependabot[bot], @derekperkins, @doeg, @fatih, @frouioui, @harshit-gangal, @malpani, @matthiasr, @mattlord, @mattrobenolt, @notfelineit, @pjambet, @rohit-nayak-ps, @rsajwani, @shlomi-noach, @simon-engledew, @systay, @utk9, @vmg, @vmogilev, @y5w, @yields

    Source code(tar.gz)
    Source code(zip)
    vitess-14.0.0-9665c18.tar.gz(488.77 MB)
    vitess-14.0.0-9665c18.x86_64.rpm(487.00 MB)
    vitess_14.0.0-9665c18_amd64.deb(488.76 MB)
  • v14.0.0-rc1(Jun 7, 2022)

    Release of Vitess v14.0.0-RC1

    Major Changes

    Command-line syntax deprecations

    Vitess has begun a transition to a new library for CLI flag parsing. In order to facilitate a smooth transition, certain syntaxes that will not be supported in the future now issue deprecation warnings when used.

    The messages you will likely see, along with explanations and migrations, are:

    "Use of single-dash long flags is deprecated"

    Single-dash usage will be only possible for short flags (e.g. -v is okay, but -verbose is not).

    To migrate, update your CLI scripts from:

    $ vttablet -tablet_alias zone1-100 -init_keyspace mykeyspace ... # old way
    

    To:

    $ vttablet --tablet_alias zone1-100 --init_keyspace mykeyspace ... # new way
    

    "Detected a dashed argument after a position argument."

    As the full deprecation text goes on to (attempt to) explain, mixing flags and positional arguments will change in a future version that will break scripts.

    Currently, when invoking a binary like

    $ vtctl --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
    

    everything after the AddCellInfo is treated by package flag as a positional argument, and we then use a sub FlagSet to parse flags specific to the subcommand. So, at the top-level, flag.Args() returns ["AddCellInfo", "--root", "/vitess/global"].

    The library we are transitioning to is more flexible, allowing flags and positional arguments to be interwoven on the command-line. For the above example, this means that we would attempt to parse --root as a top-level flag for the vtctl binary. This will cause the program to exit on error, because that flag is only defined on the AddCellInfo subcommand.

    In order to transition, a standalone double-dash (literally, --) will cause the new flag library to treat everything following that as a positional argument, and also works with the current flag parsing code we use.

    So, to transition the above example without breakage, update the command to:

    $ vtctl --topo_implementation etcd2 AddCellInfo -- --root "/vitess/global"
    $ # the following will also work
    $ vtctl --topo_implementation etcd2 -- AddCellInfo --root "/vitess/global"
    $ # the following will NOT work, because --topo_implementation is a top-level flag, not a sub-command flag
    $ vtctl -- --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
    

    New command line flags and behavior

    vttablet --heartbeat_on_demand_duration

    --heartbeat_on_demand_duration joins the already existing heartbeat flags --heartbeat_enable and --heartbeat_interval and adds new behavior to heartbeat writes.

    --heartbeat_on_demand_duration takes a duration value, such as 5s.

    The default value for --heartbeat_on_demand_duration is zero, which means the flag is not set and there is no change in behavior.

    When --heartbeat_on_demand_duration has a positive value, then heartbeats are only injected on demand, per internal requests. For example, when --heartbeat_on_demand_duration=5s, the tablet starts without injecting heartbeats. An internal module, like the lag throttle, may request the heartbeat writer for heartbeats. Starting at that point in time, and for the duration (a lease) of 5s in our example, the tablet will write heartbeats. If no other requests come in during that duration, then the tablet then ceases to write heartbeats. If more requests for heartbeats come while heartbeats are being written, then the tablet extends the lease for the next 5s following up each request. Thus, it stops writing heartbeats 5s after the last request is received.

    The heartbeats are generated according to --heartbeat_interval.

    Deprecation of --online_ddl_check_interval

    The flag --online_ddl_check_interval is deprecated and will be removed in v15. It has been unused in v13.

    Deprecation of --planner-version for vtexplain

    The flag --planner-version is deprecated and will be removed in v15. Instead, please use --planer_version.

    Online DDL changes

    Online DDL is generally available

    Online DDL is no longer experimental (with the exception of pt-osc strategy). Specifically:

    • Managed schema changes, the scheduler, the backing tables
    • Supporting SQL syntax
    • vitess strategy (online DDL via VReplication)
    • gh-ost strategy (online DDL via 3rd party gh-ost)
    • Recoverable migrations
    • Revertible migrations
    • Declarative migrations
    • Postponed migrations
    • and all other functionality

    Are all considered production-ready.

    pt-osc strategy (online DDL via 3rd party pt-online-schema-change) remains experimental.

    Throttling

    See new SQL syntax for controlling/viewing throttling for Online DDL, down below.

    ddl_strategy: 'vitess'

    --ddl_strategy now takes the value of vitess to indicate VReplication-based migrations. It is a synonym to online and uses the exact same functionality. In the future, the online term will phase out, and vitess will remain the term of preference.

    Example:

    vtctlclient ApplySchema -- --skip_preflight --ddl_strategy='vitess' --sql "alter table my_table add column my_val int not null default 0" commerce
    

    --singleton-context and REVERT migrations

    It is now possible to submit a migration with --singleton-context strategy flag, while there's a pending (queued or running) REVERT migration that does not have a --singleton-context flag.

    Support for CHECK constraints

    Online DDL operations are more aware of CHECK constraints, and properly handle the limitation where a CHECK's name has to be unique in the schema. As opposed to letting MySQL choose arbitrary names for shadow table's CHECK consraints, Online DDL now generates unique yet deterministic names, such that all shards converge onto same names.

    Online DDL attempts to preserve the original check's name as a suffix to the generated name, where possible (names are limited to 64 characters).

    Behavior changes

    • vtctlclient ApplySchema -- --uuid_list='...' now rejects a migration if an existing migration has the same UUID but with different migration_context.

    Table lifecycle

    Views

    Table lifecycle now supports views. It ensures to not purge rows from views, and does not keep views in EVAC state (they are immediately transitioned to DROP state).

    Fast drops

    On Mysql 8.0.23 or later, the states PURGE and EVAC are automatically skipped, thanks to 8.0.23 improvement to DROP TABLE speed of operation.

    Tablet throttler

    API changes

    Added /throttler/throttled-apps endpoint, which reports back all current throttling instructions. Note, this only reports explicit throttling requests (sych as ones submitted by /throtler/throttle-app?app=...). It does not list incidental rejections based on throttle thresholds.

    API endpoint /throttler/throttle-app now accepts a ratio query argument, a floating point in the range [0..1], where:

    • 0 means "do not throttle at all"
    • 1 means "always throttle"
    • any numbr in between is allowd. For example, 0.3 means "throttle in 0.3 probability", ie on a per request and based on a dice roll, there's a 30% change a request is denied. Overall we can expect about 30% of requests to be denied. Example: /throttler/throttle-app?app=vreplication&ratio=0.25

    See new SQL syntax for controlling/viewing throttling, down below.

    New Syntax

    Control and view Online DDL throttling

    We introduce the following syntax, to:

    • Start/stop throttling for all Online DDL migrations, in general
    • Start/stop throttling for a particular Online DDL migration
    • View throttler state
    ALTER VITESS_MIGRATION '<uuid>' THROTTLE [EXPIRE '<duration>'] [RATIO <ratio>];
    ALTER VITESS_MIGRATION THROTTLE ALL [EXPIRE '<duration>'] [RATIO <ratio>];
    ALTER VITESS_MIGRATION '<uuid>' UNTHROTTLE;
    ALTER VITESS_MIGRATION UNTHROTTLE ALL;
    SHOW VITESS_THROTTLED_APPS;
    

    default duration is "infinite" (set as 100 years)

    • allowed units are (s)ec, (m)in, (h)our ratio is in the range [0..1].
    • 1 means full throttle - the app will not make any progress
    • 0 means no throttling at all
    • 0.8 means on 8 out of 10 checks the app makes, it gets refused

    The syntax SHOW VITESS_THROTTLED_APPS is a generic call to the throttler, and returns information about all throttled apps, not specific to migrations

    SHOW VITESS_MIGRATIONS ... output now includes user_throttle_ratio

    This column is updated "once in a while", while a migration is running. Normally this is once a minute, but can be more frequent. The migration reports back what was the throttling instruction set by the user while it was/is running. This column does not indicate any actual lag-based throttling that takes place per production state. It only reports the explicit throttling value set by the user.

    Heartbeat

    The throttler now checks in with the heartbeat writer to request heartbeats, any time it (the throttler) is asked for a check.

    When --heartbeat_on_demand_duration is not set, there is no change in behavior.

    When --heartbeat_on_demand_duration is set to a positive value, then the throttler ensures that the heartbeat writer generated heartbeats for at least the following duration. This also means at the first throttler check, it's possible that heartbeats are idle, and so the first check will fail. As heartbeats start running, followup checks will get a more accurate lag evaluation and will respond accordingly. In a sense, it's a "cold engine" scenario, where the engine takes time to start up, and then runs smoothly.

    VDiff2

    We introduced a new version of VDiff -- currently marked as Experimental -- that executes the VDiff on tablets rather than in vtctld. While this is experimental we encourage you to try it out and provide feedback! This input will be invaluable as we improve this feature on the march toward a production-ready version. You can try it out by adding the --v2 flag to your VDiff command. Here's an example:

    $ vtctlclient --server=localhost:15999 VDiff -- --v2 customer.commerce2customer
    VDiff bf9dfc5f-e5e6-11ec-823d-0aa62e50dd24 scheduled on target shards, use show to view progress
    
    $ vtctlclient --server=localhost:15999 VDiff -- --v2 customer.commerce2customer show last
    
    VDiff Summary for customer.commerce2customer (bf9dfc5f-e5e6-11ec-823d-0aa62e50dd24)
    State: completed
    HasMismatch: false
    
    Use "--format=json" for more detailed output.
    
    $ vtctlclient --server=localhost:15999 VDiff -- --v2 --format=json customer.commerce2customer show last
    {
            "Workflow": "commerce2customer",
            "Keyspace": "customer",
            "State": "completed",
            "UUID": "bf9dfc5f-e5e6-11ec-823d-0aa62e50dd24",
            "HasMismatch": false,
            "Shards": "0"
    }
    

    :information_source: NOTE: even before it's marked as production-ready (feature complete and tested widely in 1+ releases), it should be safe to use and is likely to provide much better results for very large tables.

    For additional details, please see the RFC and the README.

    Durability Policy

    Deprecation of durability_policy Flag

    The durability policy for a keyspace is now stored in the keyspace record in the topo server. The durability_policy flag used by vtctl, vtctld, and vtworker binaries has been deprecated.

    New and Augmented Commands

    The vtctld command CreateKeyspace has been augmented to take in an additional argument called durability-policy which will allow users to set the desired durability policy for a keyspace at creation time.

    For existing keyspaces, a new command SetKeyspaceDurabilityPolicy has been added, which allows users to change the durability policy of an existing keyspace.

    If semi-sync is not being used then durability policy should be set to none for the keyspace. This is also the default option.

    If semi-sync is being used then durability policy should be set to semi_sync for the keyspace and --enable_semi_sync should be set on vttablets.

    Deprecation of Durability Configuration

    The Durability configuration is deprecated and removed from VTOrc. Instead VTOrc will find the durability policy of the keyspace from the topo server. This allows VTOrc to monitor and repair multiple keyspaces which have different durability policies in use.

    VTOrc will ignore the keyspaces which have no durability policy specified in the keyspace record. So on upgrading to v14, users must run the command SetKeyspaceDurabilityPolicy specified above, to ensure VTOrc continues to work as desired. The recommended upgrade path is to upgrade vtctld, run SetKeyspaceDurabilityPolicy and then upgrade VTOrc.


    Changelog

    Announcement

    General

    • v14 release notes: Online DDL is production ready #10310

    Bug fixes

    Backup and Restore

    • Skip repl config in restore when active reparents disabled #9675

    Cluster management

    • [grpcshim] Fix select race and move to internal package #9116
    • Use AllPrivs user for table GC transitions #9690
    • [topo/helpers] Make CopyShardReplications idempotent #9849
    • Fix the race between PromoteReplica and replication manager tick #9859
    • Propagate error from 'updateLocked' to client #10181
    • Revert semi_sync change to reintroduce enable_semi_sync and remove durability_policy #10201
    • The flag --online_ddl_check_interval belongs in vtctld #10320
    • vt/wrangler: fix deleting primary tablet if Shard does not exist #10373
    • topo: cache CellInfo along with conn #10408

    General

    • Sanitize tx serializer log & error messages #9802
    • Changes for make tools #10117
    • Sanitize log messages #10367

    Query Serving

    • Extract collation data to enable distinct aggregation #9639
    • Add QueryRowsAffected and QueryRowsReturned to vttablet metrics and deprecate QueryRowCounts #9656
    • Use the gen4 planner for queries with outer joins #9657
    • Gen4: make sure to not merge unsharded tables from different keyspaces #9665
    • Parser fix for CREATE VIEW statement #9693
    • Add collation parsing to generated columns in Create Table #9694
    • Fixed missing order/group by, limit, having in derived #9701
    • Online DDL: resubmitting a migration with same UUID retries it in case it was cancelled or failed. #9704
    • Query plan/rules: apply for multi-table statement #9747
    • Stop rewriting JoinCondition with USING #9767
    • Manage MySQL Replication Status States Properly #9853
    • Fix __sq_has_values1 error with PulloutSubquery #9855
    • OnlineDDL executor: route all VReplicationExec through single function #9861
    • fix: planner panic on derived tables sorting in query builder #9869
    • feat: add weightstring for distinct #9874
    • Fix: Sequence query to ignore reserved and transaction #9968
    • fix: dual query with exists clause having system table query in it #9969
    • fix: make concatenate and limit concurrent safe #9979
    • ApplySchema: allow-zero-in-date embedded as query comment in call to ExecuteFetchAsDba #9998
    • Fix: reserved connection retry logic when vttablet or mysql is down #10005
    • gen4: Fix sub query planning when the outer query is a dual query #10007
    • Fix parsing of bind variables in a few places #10015
    • Route explain tab plan to the proper Keyspace #10027
    • OnlineDDL: double statement validation upon submission #10065
    • Fix Gen4 only_full_group_by regression #10069
    • Fix Gen4 group/order by with derived table #10074
    • Only start SQL thread temporarily to WaitForPosition if needed #10104
    • Vitess online ddl: modify a column from textual to non-textual, ignore charset #10116
    • Fix StreamExecute in Gen4CompareV3 #10122
    • Do not mutate replication state in WaitSourcePos and ignore tablets with SQL_Thread stopped in ERS #10148
    • Fix for empty results when no shards can be found to route to #10152
    • fix: handle reserved connection reset when tx killer has locked the connection #10153
    • Do not send field query when using reserved connection #10163
    • Backwards compatible replication status to state transition #10167
    • sqlparser: Handle case sensitive AST option for table options #10191
    • Emit the ENGINE field for table options as case sensitive #10197
    • check for connectionID before adding to querylist #10212
    • Fix handling of unsigned and zerofill in parser and normalization #10220
    • Fix parsing of the foreign key constraint actions in different order #10224
    • Fix handling of VISIBLE or INVISIBLE keyword on indexes #10243
    • Fix formatting for function expressions and booleans #10255
    • fix: concatenate engine primitive #10257
    • Move all schemadiff comparisons to canonical form #10261
    • Do not cache plans that are invalid because of --no_scatter #10279
    • Fix parsing the special case convert charset logic #10288
    • Deprecate flag --online_ddl_check_interval #10308
    • Fix schema tracking issue when PRIMARY tablet changes #10335
    • Remove normalization of integral length types #10336
    • Fix failure when Unowned lookup IS NULL #10346
    • Fix for empty results when no shards can be found to route to [v3] #10360
    • Revert super_read_only config file changes #10366
    • OnlineDDL/vitess fix: convert data type to JSON #10390
    • OnlineDDL/vrepl: more error codes, spatial types #10394
    • Online DDL fix: instant is possible for VIRTUAL, not STORED #10411
    • schemadiff: column name check in CHECK constraint is case insensitive #10413
    • making log sanitization more precise #10417
    • v3: Fix issue when no routes are found from a lookup #10422
    • Tablet throttler: dynamic ThrottleMetricThreshold #10439
    • Change use_super_read_only default back to false #10448

    VReplication

    • Support VDiff across DB versions #9679
    • Externalize Lookup VIndexes properly when not stopping after copy #9771
    • Add TEXT field comparison support to evalengine (for VDiff) #9790
    • Add CHAR field comparison support to evalengine (for VDiff) #9800
    • Check for nil vschema.Table in StreamMigrator #9828
    • VPlayer: use stored/binlogged ENUM index value in WHERE clauses #9868
    • vstreamer: flatten savepoint events #9892
    • vstreamer: do not forward savepoint events #9907
    • VReplication: use strict sql_mode for Online DDL #9963
    • VStreamer: recompute table plan if a new table is encountered for the same id #9978
    • Increase max vrepl workflow source definition size from 64KiB to 16MiB #10018
    • VReplication: maintain original column case for pkCols #10033
    • Avoid deadlocks related to 0 receiver behavior #10132
    • SwitchTraffic should switch everything when no tablet_types provided #10434

    VTAdmin

    • [vtadmin] threadsafe dynamic clusters #10044

    VTorc

    • BugFix: Resolve Recoveries at the end #10286

    vtexplain

    • Case-insensitive planner-version flag in VTExplain #10086
    • fix: check that all keyspaces loaded successfully before using them #10396

    CI/Build

    Build/CI

    • Track full git sha1 string instead of the short version #9970
    • Skip CI workflows based on the list of modified files #10031
    • CI: fix flaky test via extended timeout in vrepl_stress_suite #10231
    • Fix tabletmanager_throttler flakyness: increase wait time #10271
    • remove references to MySQL and Percona 5.6 #10295

    Cluster management

    • fix endtoend onlineddl flakyness caused by --heartbeat_on_demand_duration #10236

    General

    • Upgrade main to go1.18 #9893
    • Upgrade main to go1.18.1 #10101
    • Upgrade main to go1.18.3 #10447

    Query Serving

    • More testing for vreplication unique key violation protection #9988
    • Fix to --heartbeat_on_demand_duration race condition #10247
    • OnlineDDL: adding an endtoend test to validate partial shard REVERT #10338

    Dependabot

    Examples

    • build(deps): bump gopkg.in/yaml.v2 from 2.2.5 to 2.2.8 in /examples/are-you-alive #9429

    Documentation

    Documentation

    • Update release notes with web/vtctld2 deprecation announcement and flag. #9735
    • Add help text for LegacyVtctlCommand and VtctldCommand #9837

    Examples

    • Fix commands in readme files to use double dashes for arguments #10389

    Enhancement

    Build/CI

    • Check for proto drift in the CI #9644
    • make generate_ci_workflows: mysql80 workflows for selected tests #9740
    • Add vitess/lite ubi8 images for mysql80, including for arm64 #9830

    CLI

    • [vtctldclient] Add entrypoint for GetVersion rpc #9994

    Cluster management

    • Change semi-sync information to use the parameter passed and deprecate enable_semi_sync #9725
    • Filter candidates which cannot make forward progress after ERS #9765
    • [vtctldserver] Add locking checks to Delete{Keyspace,Shard} #9777
    • [topo] ShardReplicationFix typed responses #9876
    • tablet lag throttler: small API improvements #10045
    • OnlineDDL: skip GetSchema where possible #10107
    • Refresh ephemeral information before cluster operations in VTOrc #10115
    • On demand heartbeats via --heartbeat_on_demand_duration, used by the tablet throttler #10198
    • Adds DurabilityPolicy to the KeyspaceInfo in the topo and the associated RPCs #10221
    • Refactor Durability Policy implementation and usage to read the durability policy from the keyspace #10375
    • Use Durability Policy in the topo server in VTOrc and deprecate Durability config #10423

    Examples

    • Add consul-topo for local example #9806
    • Add vtadmin to local example by default #10430
    • Halve request and double limit for RAM in the example cluster #10450

    General

    • Binlog event parsing: better analysis ; support for semi-sync #9596
    • Support views in Online DDL #9606
    • [VEP-4, phase 1] Flag Deprecation Warnings #9733
    • Table lifecycle: support fast DROP TABLE introduced in MySQL 8.0.23 #9778
    • Flavor capabilities: MySQL GR and more #10451

    Query Serving

    • Harmonize error codes for pool timeouts expiring across all pools #9483
    • Store innodbRowsRead in a Counter instead of a Gauge #9609
    • Improve performance of information schema query. #9632
    • OnlineDDL: reject duplicate UUID with different migration_context #9637
    • Reduce the number of reserved connections when setting system variables #9641
    • OnlineDDL: 'vitess' strategy, synonym for 'online' #9642
    • Gen4: Rework how aggregation is planned #9643
    • Support for ALTER TABLE ... PARTITION BY ... #9683
    • OnlineDDL: force (once) vreplication's WithDDL to run before running a vitess migration #9702
    • Push projection to union #9703
    • Make gen4 the default planner for release 14 #9710
    • Cleanup of parsing of Partitions and additional parsing support #9712
    • Introducing schemadiff, a declarative diff for table/view CREATE statements #9719
    • Online DDL vitess migration's cut-over: query buffering #9755
    • Online DDL declarative migrations now use schemadiff, removing tengo #9772
    • Online DDL: ready_to_complete hint column #9813
    • Add parsing support for prepare statements #9818
    • Online DDL: vitess migrations cut-over to have zero race conditions #9832
    • Add parsing support for Trim grammar function #9834
    • Add parsing support for LATERAL keyword #9843
    • Add parsing support for JSON Utility Functions #9850
    • Remove the Gateway interface #9852
    • Gen4: Add UPDATE planning #9871
    • Add parsing support for JSON_TABLE() #9879
    • Add parsing support for JSON value creators functions #9880
    • fix: allow multiple distinct columns on single shards #9940
    • Add parsing support for Json schema validation functions #9971
    • Add parsing support for JSON Search Functions #9990
    • deps: upgrade grpc and protobuf #10024
    • feat: allow more complex expressions to be used in outer queries #10034
    • schemadiff: load, normalize, validate and compare schemas #10048
    • Add parsing support for JSON value modification and attribute functions #10062
    • Always reset reserved shard session on CODE_UNAVAILABLE #10063
    • Adds parsing for NOW in DEFAULT clause #10085
    • Gen4 feature: optimize OR queries to IN #10090
    • Relax singleton-context constraint for pending non-singleton-context reverts #10114
    • Add parsing support for Partition Definitions #10127
    • feat gen4: allow pushing aggregations inside derived tables #10128
    • schemadiff: support for RangeRotationStrategy diff hint #10131
    • Avoid enabling reserved connection on show query #10138
    • Add support for non aggregated columns in OrderedAggregate #10139
    • schemadiff: CanonicalStatementString(), utilize sqlparser.CanonicalString() #10142
    • feat: make gen4 plan subsharding queries better #10151
    • schemadiff: EntityDiff has a 'Entities() (from Entity, to Entity)' function #10161
    • Introducing a SQL syntax to control and view Online DDL throttling #10162
    • feat: optimize EXISTS queries through AST rewriting #10174
    • schemadiff: logically Apply() a diff to a CreateTable, CreateView or to a Schema #10183
    • Using LCS to achieve shortest column reordering list #10184
    • enhancement: handle advisory locks #10186
    • schemadiff: normalize() table. Set names to all keys #10188
    • schemadiff: validate() table structure at the end of apply() #10189
    • schemadiff: schema is immutable through Apply() #10190
    • VReplication: use PK equivalent (PKE) if no PK found #10192
    • Add parsing support for Window Functions #10199
    • schemadiff: normalize table options case #10200
    • Allow updating unowned lookup vindex columns. #10207
    • partial dml execution logic ehancement #10223
    • gen4: JOIN USING planning #10226
    • schemadiff: partitions validations #10229
    • Add parsing support for Subpartition Definitions #10232
    • schemadiff: analyze and apply ADD PARTITION and DROP PARTITION statements #10234
    • Add convenience functions for working with schemadiff objects #10238
    • Add parsing support for expressions in index definitions #10240
    • Update the parsing support for JSON_VALUE #10242
    • schemadiff: support functional indexes #10244
    • Fix formatting of character set annotation on string literals #10245
    • schemadiff: use ParseStrictDDL whenever expecting a DDL statement #10246
    • Fix parsing of encryption attribute for create database #10249
    • Add support for parsing MATCH in a foreign key definition #10250
    • Add parsing support for additional index and table options #10251
    • Handle additional DDL in column definitions #10253
    • evalengine: CASE expressions #10262
    • vt/sqlparser: support SRID #10267
    • Improve dealing with check constraints #10268
    • Add parsing and schemadiff support for ALTER TABLE ALTER CHECK #10269
    • Add unary utf8mb3 charset marker #10274
    • Allow updating the subsequent diff #10289
    • Improve the formatting for partitioning #10291
    • Add additional normalization rules for schemadiff #10296
    • schemadiff: RangeRotationDistinctStatements partitioning hint #10309
    • Remove internal savepoint post query execution #10311
    • feat: use column offsets when HAVING predicate has aggregation #10312
    • Improve parser and schemadiff support for ALTER TABLE #10313
    • Online DDL: fast (and correct) RANGE PARTITION operations, 1st iteration #10315
    • Add support for additional encoding configurations #10321
    • Add parsing support for Regular Expressions #10322
    • schemadiff: validate columns referenced by generated columns #10328
    • schemadiff: case insensitive col/index/partition name comparison #10330
    • [gen4] More aggregation support #10332
    • Add better int normalization #10340
    • Additional schemadiff improvements for indexes #10345
    • Structured schemadiff errors #10356
    • Add validation of check constraints #10357
    • schemadiff: validate columns referenced by FOREIGN KEY #10359
    • Aggregation on top of LIMIT #10361
    • Add support for parsing ARRAY in cast / convert operations #10362
    • Online DDL: support for CHECK CONSTRAINTS #10369
    • schemadiff: constraint name diff hints strategies #10380
    • Replace 'CREATE TABLE LIKE' with programmatic copy #10381
    • Streaming implementation of Projection engine primitive #10384
    • feat: do not use lookup vindexes for IS NULL predicates #10388
    • Online DDL: support INSTANT DDL special plan, flag protected (flag undocumented for now) #10402
    • feat: make v3 not plan IS NULL queries against a lookup vindex #10412
    • Online DDL: support Instant DDL for change of column's DEFAULT #10414
    • Parsing Support for XML functions #10438
    • Formalize MySQL capabilities by flavor/version #10445

    TabletManager

    • messaging: support vt_message_cols to limit cols #9670
    • Table lifecycle: support views #9776

    VReplication

    • OnlineDDL: delete _vt.vreplication entry as part of CLEANUP operation #9638
    • Tablet server/VReplication: use information_schema.columns to get the list of columns instead of "select * from table" #9794
    • Add --drop_constraints to MoveTables #9904
    • Support throttling vstreamer copy table work on source tablets #9923
    • Improve escape handling for vindex materializer #9929
    • VReplication: fail on unrecoverable errors #9973
    • Prefer using REPLICA tablets when selecting vreplication sources #10040
    • Improve Tablet Refresh Behavior in VReplication Traffic Switch Handling #10058
    • VReplication: use db_filtered user for vstreams #10080
    • VStream API: allow cells to be specified for picking source tablets to stream from #10294
    • MoveTables: adjust datetimes when importing from non-UTC sources into UTC targets #10102
    • Vdiff2: initial release #10382

    VTAdmin

    • [vtadmin-web] Add REACT_APP_READONLY_MODE flag #9731
    • [vtadmin] custom discovery resolver #9977
    • [vtadmin] grpc healthserver + channelz #10038
    • [Dynamic Clusters] No Duplicates #10043
    • [vtadmin] full dynamic cluster config #10071
    • [vtadmin] non-blocking resolver #10205
    • [vtadmin] tablet topopools + other cleanup #10325
    • [vtadmin] cluster Closer implementation + dynamic Cluster bugfix #10355

    VTCombo

    • Add flag to vtcombo to load a json-encoded test topology file #9633

    VTorc

    • Revocation phase using durability policies #9659
    • VTOrc: Refactor and reload of ephemeral information for remaining recovery functions #10150
    • vtorc: more auditing in GracefulPrimaryTakeover #10285

    vttestserver

    • vtcombo: Add the ability to specify routing rules in test topology #9695

    web UI

    • [vtctld] Add 'enable_vtctld_ui' flag to vtctld to explicitly enable (or disable) the vtctld2 UI #9614

    Feature Request

    Cluster management

    • [vtctldserver] Migrate Backup RPC to vtctldserver/client #9798
    • [vtctldclient] Add entrypoint for ApplySchema #9829
    • [vtctldserver] Migrate BackupShard RPC #9857
    • [vtctldserver] Migrate RemoveBackup RPC #9865
    • [vtctldserver] Migrate RestoreFromBackup RPC #9873
    • [vtctldserver] Migrate ShardReplicationFix RPC #9878
    • [vtctldserver] Migrate SourceShard{Add,Delete} RPCs #9886
    • [vtctldserver] ExecuteFetchAs{App,DBA} #9890
    • [vtctldserver] Migrate ShardReplication{Add,Remove} RPCs #9899
    • [vtctldserver] Migrate GetPermissions PR#9903 #10013

    Observability

    • [vtadmin] Add prom metrics endpoint support #10334

    Query Serving

    • mysql: allow parsing the 'info' field of MySQL packets #9651
    • Add support for insert using select #9748
    • Add parsing support for Multi INDEX HINTS #9811

    VTAdmin

    • [VTAdmin] Tablet Actions (vtctld2 Parity) #9601
    • [VTAdmin][vtctldserver] Keyspace Functions #9667
    • Allow --no-rbac flag that allows users to not pass rbac config #9972
    • [vtadmin] grpc dynamic clusters #10050
    • [vtadmin] schema cache #10120
    • [vtadmin] cell apis #10227
    • [vtadmin] reload schema #10300
    • [vtadmin] reparenting actions #10351

    Internal Cleanup

    Build/CI

    • Remove usage of additional uuid package #10307

    CLI

    • [vtctl] Deprecate query commands #9934
    • [vtctl] Deprecate Throttler RPCs #9962
    • [vtadmin] Remove deprecated dynamic cluster flag #10067
    • Update vtctl help output to use double dashes for long flag names #10405

    Cluster management

    • Move deprecation warning to just before we actually invoke the legacy command path #9787
    • Online DDL: cleanup old vtctld+topo flow #9816
    • Rename ApplySchemaRequest.request_context=>migration_context #9824
    • rename master_alias to primary_alias in reparent_journal table #10098
    • Use the HealthCheckImpl in vtctld instead of the LegacyHealthCheck #10254
    • Cleanup: removed legacy, unused code (online DDL paths in topo) #10379

    Examples

    • Examples: fix warnings #9875
    • Migrate remaining vtctldclient commands in local example #9894

    General

    • Support sanitizing potentially sensitive information in log messages #9550
    • [VEP-4, phase 1] Flag cleanup in go/cmd #9896

    Query Serving

    • Delete discovery gateway #9500
    • Remove queryserver-config-terse-errors impact on log messages #9634
    • Split aggregation in scalar and ordered aggregation #9810
    • Minor fix requested by linter #9989
    • remove set_var session field from proto #10135

    TabletManager

    • delete RPCs that were deprecated in 13.0 #10099

    VReplication

    • remove changeMasterToPrimary, we no longer need it #10100
    • Ensure all legacy sharding commands are clearly deprecated #10281

    VTAdmin

    • [vtadmin] threadsafe vtsql #10000
    • [vtsql] Remove vtsql.PingContext check from dial calls #10037
    • [vtadmin] Refactor api.getSchemas => cluster.GetSchemas #10108
    • [vtadmin] Remove explicit Dial-ing for vtctldclient proxy #10233
    • [vtadmin] Removing explicit Dial-ing for vtsql proxy #10237
    • [vtadmin] proper alias types for tablet RPCs #10290
    • [vtadmin] Remove mutex, now that vtexplain has no global state #10387
    • [vtadmin] Rename reparent-related RPCs, endpoints, methods, RBAC actions #10404

    vtctl

    • Fixes #8277: vtctld logs leak pii http headers #9669

    vtexplain

    • replace planner-version with planner_version in vtexplain #10420

    Performance

    Schema Tracker

    • Improve performance of the DetectSchemaChange query #10416

    Testing

    Build/CI

    • More aggressive tests for vitess migration cut-over #9956
    • fix tablegc flaky test #10230

    CLI

    • [VEP-4, part 1] Flag cleanup in go/mysql and some endtoend tests #9910
    • [VEP4, phase 1] Update flags in endtoend/recovery #9911
    • [VEP4, phase 1] Update flags in endtoend/schemadiff #9912
    • [VEP4, phase 1] Update flags and imports for endtoend/{sharded,sharding} #9913
    • [VEP 4, phase 1] Cleanup flags in endtoend/vault (and some comments in endtoend/tabletmanager) #9916
    • [VEP4, phase 1] fix flags in tests endtoend/versionupgrade #9917
    • [VEP4, phase 1] [endtoend/vreplication] update flags in tests #9918
    • [VEP4, phase 1] [endtoend/vtcombo] fix flags in tests #9919
    • [VEP4, phase1] Cleanup flags in endtoend/vtgate and endtoend/vtorc #9930
    • [VEP4, phase1] fix flags in tests in endtoend/worker and strings in go/trace #9931

    General

    • test: use T.TempDir to create temporary test directory #10433

    Query Serving

    • vtexplain: added multicol test to show subshard routing #9697
    • Compare Vitess against MySQL in E2E tests #9965
    • Make the tests explicit for the SET variables for MySQL 5.7 #10263

    The release includes 1065 commits (excluding merges)

    Thanks to all our contributors: @FancyFane, @GuptaManan100, @Juneezee, @K-Kumar-01, @Phanatic, @ajm188, @akenneth, @aquarapid, @arthurschreiber, @brendar, @cuishuang, @dasl-, @dbussink, @deepthi, @dependabot[bot], @derekperkins, @doeg, @fatih, @frouioui, @harshit-gangal, @malpani, @matthiasr, @mattlord, @mattrobenolt, @notfelineit, @pjambet, @rohit-nayak-ps, @rsajwani, @shlomi-noach, @simon-engledew, @systay, @utk9, @vmg, @vmogilev, @y5w, @yields

    Source code(tar.gz)
    Source code(zip)
    vitess-14.0.0-rc1-7821f05.tar.gz(487.90 MB)
    vitess-14.0.0_rc1-7821f05.x86_64.rpm(486.13 MB)
    vitess_14.0.0-rc1-7821f05_amd64.deb(487.91 MB)
  • v12.0.4(May 20, 2022)

    Release of Vitess v12.0.4

    Major Changes

    Cluster management

    A race condition #9819 that happens when running PlannedReparentShard was fixed through #9859.

    Query Serving

    Two major bugs on UNION are fixed via this release. The first one, #10257, solves an issue around the UNION executor, where the execution context was not handled properly in a concurrent environment. The second one, #9945, was detected by one of our UNION tests becoming flaky in the CI, it got solved by #9979, which focuses on improving the concurrency of UNION executions.

    A panic when ordering results in descending order on a hash vindex is also fixed. The original issue can be found here #10019.

    Changelog

    Bug fixes

    Cluster management

    • Fix the race between PromoteReplica and replication manager tick #9859

    Query Serving

    • Route explain table plan to the proper Keyspace #10028
    • Multiple fixes to UNION planning and execution #10344

    The release includes 5 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @frouioui, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.4-c462e63.tar.gz(400.65 MB)
    vitess-12.0.4-c462e63.x86_64.rpm(398.94 MB)
    vitess_12.0.4-c462e63_amd64.deb(400.65 MB)
  • v13.0.1(Apr 14, 2022)

    Release of Vitess v13.0.1

    Changelog

    Bug fixes

    Cluster management

    • Fix the race between PromoteReplica and replication manager tick #9859

    Query Serving

    • Fix __sq_has_values1 error with PulloutSubquery #9864
    • Fix planner panic on derived tables sorting in query builder #9959
    • Fix make concatenate and limit concurrent safe #9981
    • Fix reserved connection retry logic when vttablet or mysql is down #10005
    • Fix Gen4 sub query planning when the outer query is a dual query #10007
    • Fix parsing of bind variables in a few places #10015
    • Fix route explain tab plan to the proper Keyspace #10029
    • Fix Sequence query to ignore reserved and transaction #10054
    • Fix dual query with exists clause having system table query in it #10055
    • Fix Gen4 only_full_group_by regression #10079

    VReplication

    • VPlayer use stored/binlogged ENUM index value in WHERE clauses #9868

    CI/Build

    Security

    • Upgrade to go1.17.9 #10088 The go1.17.9 version, released 2022-04-12, includes security fixes to the crypto/elliptic and encoding/pem packages, as well as bug fixes to the linker and runtime). More information here.

    Enhancement

    Query Serving

    • Fix allow multiple distinct columns on single shards #10047
    • Fix add parsing for NOW in DEFAULT clause #10085

    The release includes 30 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @systay, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-13.0.1-f4a77b6.tar.gz(463.28 MB)
    vitess-13.0.1-f4a77b6.x86_64.rpm(461.63 MB)
    vitess_13.0.1-f4a77b6_amd64.deb(463.29 MB)
  • v13.0.0(Feb 22, 2022)

    Release of Vitess v13.0.0

    Major Changes

    Vitess now has native support for MySQL collations

    When using the gen4 planner, Vitess is now capable of performing collation-aware string comparisons in the vtgates. This improves the performance and reliability of several query plans that were previously relying on a debug-only SQL API in MySQL to perform these comparisons remotely. It also enables new query plans that were previously not possible.

    A full list of the supported collations can be found in the Vitess documentation.

    The native Evaluation engine in vtgate has been greatly improved

    The SQL evaluation engine that runs inside the vtgates has been rewritten mostly from scratch to more closely match MySQL's behavior. This allows Vitess to execute more parts of the query plans locally, and increases the complexity and semantics of the SQL expressions that can be used to perform cross-shard queries.

    vttablet -use_super_read_only flag now defaults to true

    The default value used to be false. What this means is that during a failover, we will set super_read_only on database flavors that support them (MySQL 5.7+ and Percona 5.7+). In addition, all Vitess-managed databases will be started with super-read-only in the cnf file. It is expected that this change is safe and backwards-compatible. Anyone who is relying on the current behavior should pass -use_super_read_only=false on the vttablet command line, and make sure they are using a custom my.cnf instead of the one provided as the default by Vitess.

    vtgate -buffer_implementation now defaults to keyspace_events

    The default value used to be healthcheck. The new keyspace_events implementation has been tested in production with good results and shows more consistent buffering behavior during PlannedReparentShard operations. The keyspace_events implementation utilizes heuristics to detect additional cluster states where buffering is safe to perform, including cases where the primary may be down. If there is a need to revert back to the previous buffer implementation, ensure buffering is enabled in vtgate and pass the flag -buffer_implementation=healthcheck.

    ddl_strategy: -postpone-completion flag

    ddl_strategy (either @@ddl_strategy in VtGate or -ddl_strategy in vtctlclient ApplySchema) supports the flag -postpone-completion

    This flag indicates that the migration should not auto-complete. This applies for:

    • any CREATE TABLE
    • any DROP TABLE
    • ALTER table in online strategy
    • ALTER table in gh-ost strategy

    Note that this flag is not supported for pt-osc strategy.

    Behavior of migrations with this flag:

    • an ALTER table begins, runs, but does not cut-over.
    • CREATE or DROP migrations are silently not even scheduled

    alter vitess_migration ... cleanup

    A new query is supported:

    alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' cleanup
    

    This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As a reminder, once a migration's artifacts are cleaned up, the migration is no longer revertible.

    alter vitess_migration ... complete

    A new query is supported:

    alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete
    

    This command indicates that a migration executed with -postpone-completion is good to complete. Behavior:

    • For running ALTERs (online and gh-ost) which are ready to cut-over: cut-over imminently (though not immediately
      • cut-over depends on polling interval, replication lag, etc)
    • For running ALTERs (online and gh-ost) which are only partly through the migration: they will cut-over automatically when they complete their work, as if -postpone-completion wasn't indicated
    • For queued CREATE and DROP migrations: "unblock" them from being scheduled. They'll be scheduled at the scheduler' s discretion. there is no guarantee that they will be scheduled to run immediately.

    vtctl/vtctlclient ApplySchema: ALTER VITESS_MIGRATION

    vtctl ApplySchema now supports ALTER VITESS_MIGRATION ... statements. Example:

    $ vtctl ApplySchema -skip_preflight -sql "alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete" commerce
    

    vtctl/vtctlclient ApplySchema: allow zero in date

    vtctl/vtctlclient ApplySchema now respects -allow-zero-in-date for direct strategy. For example, the following statement is now accepted:

    vtctlclient ApplySchema -skip_preflight -ddl_strategy='direct -allow-zero-in-date' -sql "create table if not exists t2(id int primary key, dt datetime default '0000-00-00 00:00:00')" commerce
    

    vtctl/vtctlclient ApplySchema -uuid_list

    vtctlient ApplySchema now support a new optional -uuid_list flag. It is possible for the user to explicitly specify the UUIDs for given migration(s). UUIDs must be in a specific format. If given, number of UUIDs must match the number of DDL statements. Example:

    vtctlclient OnlineDDL ApplySchema -sql "drop table t1, drop table t2" -uuid_list "d08f0000_51c9_11ec_9cf2_0a43f95f28a3,d08f0001_51c9_11ec_9cf2_0a43f95f28a3" commerce
    

    Vitess will assign each migration with given UUID in order of appearance. It is the user's responsibility to ensure given UUIDs are globally unique. If the user submits a migration with an already existing UUID, that migration never gets scheduled nor executed.

    vtctl/vtctlclient ApplySchema -migration_context

    -migration_context flag is synonymous to -request_context. Either will work. We will encourage use of -migration_context as it is more consistent with output of SHOW VITESS_MIGRATIONS ... which includes the migration_context column.

    vtctl/vtctlclient OnlineDDL ... complete

    Complementing the alter vitess_migration ... complete query, a migration can also be completed via vtctl or vtctlclient:

    vtctlclient OnlineDDL <keyspace> complete <uuid>
    

    For example:

    vtctlclient OnlineDDL commerce complete d08ffe6b_51c9_11ec_9cf2_0a43f95f28a3
    

    vtctl/vtctlclient OnlineDDL -json

    The command now accepts an optional -json flag. With this flag, the output is a valid JSON listing all columns and rows.

    vtadmin-web updated to node v16.13.0 (LTS)

    Building vtadmin-web now requires node >= v16.13.0 (LTS). Upgrade notes are given in https://github.com/vitessio/vitess/pull/9136.

    PlannedReparentShard for cluster initialization

    For setting up the cluster and electing a primary for the first time, PlannedReparentShard should be used instead of InitShardPrimary.

    InitShardPrimary is a forceful command and copies over the executed gtid set from the new primary to all the other replicas. So, if the user isn't careful, it can lead to some replicas not being setup correctly and lead to errors in replication and recovery later. PlannedReparentShard is a safer alternative and does not change the executed gtid set on the replicas forcefully. It is the preferred alternate to initialize the cluster.

    If using a custom init_db.sql that omits SET sql_log_bin = 0, then InitShardPrimary should still be used instead of PlannedReparentShard.

    Durability Policy flag

    A new flag has been added to vtctl, vtctld and vtworker binaries which allows the users to set the durability policies.

    If semi-sync is not being used then -durability_policy should be set to none. This is also the default option.

    If semi-sync is being used then -durability_policy should be set to semi_sync and -enable_semi_sync should be set in vttablets.

    Incompatible Changes

    Error message change when vttablet row limit exceeded:

    • In previous Vitess versions, if the vttablet row limit (-queryserver-config-max-result-size) was exceeded, an error like:
      ERROR 10001 (HY000): target: unsharded.0.master: vttablet: rpc error: code = ResourceExhausted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      would be reported to the client.
      

      To avoid confusion, the error mapping has been changed to report the error as similar to:

      ERROR 10001 (HY000): target: unsharded.0.primary: vttablet: rpc error: code = Aborted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      instead
      
    • Because of this error code change, the vttablet metric: vttablet_errors{error_code="ABORTED"} will be incremented upon this type of error, instead of the previous metric: vttablet_errors{error_code="RESOURCE_EXHAUSTED"}
    • In addition, the vttablet error message logged is now different. Previously, a (misleading; due to the PoolFull) error was logged:
      E0112 09:48:25.420641  278125 tabletserver.go:1368] PoolFull: Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      

      Post-change, a more accurate warning is logged instead:

      W0112 09:38:59.169264   35943 tabletserver.go:1503] Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      
    • If you were using -queryserver-config-terse-errors to redact some log messages containing bind vars in 13.0-SNAPSHOT, you should now instead enable -sanitize_log_messages which sanitizes all log messages containing sensitive info

    Column types for textual queries now match MySQL's behavior

    The column types for certain queries performed on a vtgate (most notably, those that SELECT system variables) have been changed to match the types that would be returned if querying a MySQL instance directly: textual fields that were previously returned as VARBINARY will now appear as VARCHAR.

    This change should not have an impact on MySQL clients/connectors for statically typed programming languages, as these clients coerce the returned rows into whatever types the user has requested, but clients for dynamic programming languages may now start returning as "string" values that were previously returned as "bytes".

    Deprecations

    vtgate -gateway_implementation flag is deprecated (and ignored)

    Support for discoverygateway is being dropped. tabletgateway is now the only supported implementation. Scripts using this flag should be updated to remove the flag as it will be deleted in the next release.

    web/vtctld2 is deprecated and can optionally be turned off

    The vtctld2 web interface is no longer maintained and is planned for removal in Vitess 16. Motivation for this change and a roadmap to removing the web/vtctld2 codebase is given in https://github.com/vitessio/vitess/issues/9686.

    Vitess operators can optionally disable the vtctld2 web interface ahead of time by calling the vtctld process with the flag enable_vtctld_ui=false. For more details on this flag, see https://github.com/vitessio/vitess/pull/9614.


    Changelog

    Bug fixes

    Backup and Restore

    • vtctl: Check ShardReplicationStatuses error and make sure they are well returned #8966
    • Skip replication config in restore when active reparents are disabled #9688
    • Fix semi-sync for Backup and RestoreFromBackup commands by calling SetReplicationSource RPC #9730

    Cluster management

    • Fix vtctlclient DeleteCellInfo(force=true) command with downed local topo #9081
    • Fix race condition when rapidly starting/stopping health check #9152
    • Topo write error in change tablet type #9157
    • Default build info tablet tags to off #9187
    • Replication manager should not start health ticks after calling Promote Replica #9274
    • Fix error checking in ERS #9330
    • Fix for context cancellation in ERS #9503

    General

    • Avoid some pollution of CLI argument namespace across vtgate and vttablet #8931
    • Update to the latest Tengo version #9110

    Orchestrator

    • Set orc maintenance mode on tablet that is being promoted by PRS #8859

    Query Serving

    • Fix for schema tracker not recognizing RENAME for schema reloading #8963
    • Handle sql_mode differently for new value change #9014
    • Fix savepoint support with reserved connections #9030
    • Use a DBA user pool for the heartbeat writer #9033
    • Handle single column vindex correctly with multi-columns in Gen4 #9060
    • Remove tablet healthcheck cache record on error #9106
    • Fix request buffering while reparenting #9112
    • Ensure that the hex query predicates are normalized for planner cache #9118
    • Fix table/sizes query, to allow vttablet schema engine to see partitioned tables #9188
    • Remove keyspace from the query before sending it to the shards #9190
    • Make sure to copy bindvars when using them concurrently #9191
    • Fix owned vindex query to keep alias in table expression #9244
    • Use decoded hex string when calculating the keyspace ID #9277
    • Fix show statement in prepare call #9280
    • Avoid parenthesis around default null in BLOB, TEXT, GEOMETRY, and JSON column #9301
    • Allow unowned lookup_unique vindex columns to be null #9302
    • Estimate replica lag when the number of seconds behind from mysqld is unknown #9308
    • Clear existing keyspace schema in vtgate before [re]loading it #9437
    • Fix AST copy issue in the Gen4Fallback planner #9487
    • Fix a bug in the Gen4 planner for join query pushing output column on the left-hand side #9521
    • Extract collation data to enable distinct aggregation #9645
    • Use the gen4 planner for queries with outer joins #9671
    • Gen4: make sure not to merge unsharded tables from different keyspaces #9676
    • Fixed missing order/group by, limit, having in derived #9707

    TabletManager

    • Prevent Race Conditions Between Tablet Deletes and Updates #9237

    VReplication

    • Fix boolean parameter order in DropSources call for v2 flows #9175
    • VReplication Workflow Create: Check for Copying state while waiting for streams to start #9206
    • MoveTables: update vschema while moving tables with autoincrement from sharded to unsharded #9288
    • Schema Engine now accountable for Online DDL swapped tables #9324
    • Take MySQL Column Type Into Account in VStreamer #9331
    • Allow source columns to be specified multiple times in a vreplication rule filter #9335
    • Fixed case-sensitive TabletType in SwitchTraffic #9440
    • Use LOCK TABLES during SwitchWrites to prevent lost writes #9481
    • VReplication Workflow: manage SQL_MODE as MySQL does in mysqldump #9505
    • Routing Rules: Improved escaping of reserved and invalid names for tables and keyspaces #9522
    • Ignore internal tables in MoveTables and Reshard Workflows #8992,#9578

    VTCombo

    • vtcombo: CREATE DATABASE creates shards as specified by the topology #9130

    CI/Build

    Build/CI

    • Remove Percona5.6 Unit Tests for Vitess 13.0 #9067
    • Addition of upgrade downgrade testing to the test suite #9300, #9473, #9501
    • Re-enable default Make behavior of failing target if any command fails #9319
    • ubuntu-latest now uses MySQL 8.0.26, let us override it with latest 8.0.x #9368
    • Add vtadmin binary to release and Docker images #9405

    VTAdmin

    • Enforce node + npm versions with .npmrc #9336

    vtexplain

    • Correct handling of column types for queries involving multiple tables #9260

    Dependabot

    Build/CI

    • Upgrade go-proxyproto to v0.6.1 to fix vulnerabilities #9425

    Java

    • Bump log4j-core from 2.15.0 to 2.16.0 in /java #9387
    • Bump log4j-api from 2.16.0 to 2.17.1 in /java #9423
    • Updates all major Java dependencies to latest #9599

    VTAdmin

    • Bump tmpl from 1.0.4 to 1.0.5 in /web/vtadmin #9278

    Enhancement

    Authn/z

    • Add support for TLS certification revocation list (CRL) files #8807

    Backup and Restore

    • Set super_read_only off during restore (achieves same as PR #8929) #9240

    Cluster management

    • Support -allow-zero-in-date in ApplySchema 'direct' strategy #9273
    • Remove legacy OnlineDDL code deprecated in earlier version #8971
    • Add version-info to tablet tags in the topo #8973
    • Add default collation to VTGate and VTTablet #9097
    • Use relay log position in PRS while choosing the new primary #9270
    • vtctl OnlineDDL supports 'complete' command #9298
    • vtctl ApplySchema accepts 'ALTER VITESS_MIGRATION...' statements #9303
    • Make the heartbeat writer use 2 pools #9320
    • New vtctl ApplySchema flag -uuid_list #9325
    • Fix ERS to be used for initialization as well #9511
    • Fix ERS to work when the primary candidate's replication is stopped #9512
    • Add 'enable_vtctld_ui' flag to vtctld to explicitly enable (or disable) the vtctld2 UI (backport) #9713

    Examples

    • Update operator.yaml to v2.5.0 #9016

    General

    • vitessdriver: add support for DistributedTx #9451

    Observability

    • Modify terse errors to also redact errors in logs #9094

    Query Serving

    • Initial implementation of partitions (Create statement) #8691
    • Add parsing for common table expressions using WITH clause #8918
    • Support SQL SELECT LIMIT #8944
    • Support for union in subquery in Gen4 #8948
    • Support for ordering using derived table columns in Gen4 #8961
    • Add planner-version flag to vtexplain #8979
    • Rewrite NOT expressions for cleaner plans #8987
    • Minimize logging of errors when loading keyspace with disabled schema tracking #8989
    • Support for ordering on vindex function and straight_join queries to be planned by ignoring the hint as warning in Gen4 #8990
    • Improve internal subquery dependencies update in Gen4 #8998
    • Added tokenizer support for NCHAR_STRING #9006
    • Query timeout in comment directive in Gen4 #9008
    • Added brackets in default value if column is of type TEXT, BLOB, GEOMETRY or JSON #9011
    • Addition of the filter primitive and planning in Gen4 #9017
    • Initial integration of the collation module in Gen4 #9018
    • Support extract function in the SQL parser #9029
    • Add collation name to the type definition in Gen4 #9038
    • Support semi-join in Gen4 #9039
    • Changed Collations and Expression grammar #9075
    • Support transactions with OLAP workload #9115
    • Use of advisory lock in prepare queries #9129
    • Add hash join primitive and planning in Gen4 #9140
    • Respect --allow-zero-in-date in CREATE and in declarative migrations #9142
    • Improve merging of DBA queries in Gen4 #9183
    • Evaluate REVERTibility of a migration in OnlineDDL #9232
    • Improve group by queries planning when they use a sharding key #9243
    • Make sure VTGate and VTTablet collations are matching #9248
    • Support filtering on derived union system table with star projection #9263
    • MySQL: Better handling of query failures within a transaction to match the MySQL behavior #9269
    • Add Tuples and InOp implementation to EvalEngine #9281
    • Use evalengine for Vindex lookups in route primitive #9291
    • Adds parsing support for string introducers #9309
    • Multi-column vindex support for DML #9338
    • Add list (IN()) support to vindex functions #9426
    • Fix parsing of table names that start with a number #9456
    • Make the MaxWaiters setting configurable for the OLAP pool #9484
    • Add support for health service in gRPC server #9528
    • Filter after left join #9531
    • Add Information_Schema to evalengine #9536
    • Faster and safer unsharded query planning #9542
    • Add support for PLANNER SQL comment #9545
    • MySQL: use UTF8MB4 consistently as the connection charset #9558
    • Do not auto retry after "server lost" errors (errno 2013) in vttablet #9009
    • Online DDL: migration with same context and SQL now considered duplicate and silently marked as complete #9107
    • Support ALTER VITESS_MIGRATION ... CLEANUP statement #9160
    • OnlineDDL: implementing -postpone-completion, ALTER VITESS_MIGRATION ... COMPLETE #9171
    • OnlineDDL: support concurrent REVERT migrations #9192
    • VTGate: change default query buffer implementation (used to avoid client errors during state transitions such as PlannedReparentShard) #9359 #9360
    • Fix misleading error mapping for row limit error #9448
    • Add -sanitize_log_messages tablet flag to sanitize all log messages which may contain sensitive info and remove -queryserver-config-terse-errors impact on log messages #9636

    TabletManager

    • Have vttablet use super_read_only by default when a tablet becomes a replica #9312
    • MySQL: Add ConnectionReady callback to Handler interface #9496
    • MySQL: Add UnimplementedHandler struct #9526

    VReplication

    • Support VStream with keyspaces_to_watch #8988
    • VEvent: Add Keyspace/Shard properties to proto #9386
    • Initial refactoring of the 'vstream * from' functionality in vtgate #9392
    • Add flag to keep routing rules in v2 vrepl workflows #9441
    • SwitchTraffic: check vreplication lag before switching #9538

    VTAdmin

    • [vtadmin] viper config support #9154
    • [VTAdmin] Update react-scripts, postcss #9493

    VTorc

    • Have a common EmergencyReparentShard (ERS) implementation for both VtOrc and Vtctl #8492
    • Use PlannedReparentShard (PRS) in VtOrc while doing Graceful Primary Takeover #9258
    • Make PRS use Durability Rules #9259
    • Use PRS in Vtorc for electing new primary #9409

    vtctl

    • ListAllTablets: improve efficiency within vitess (fewer topo calls) and for caller (keyspace and tablet type filters) #9560
    • vtctl OnlineDDL show: export all columns, support -json #9568
    • vtctl ApplySchema: introduce -migration_context flag, synonym to -request_context #9572

    vttestserver

    • Add option to run vtcombo and vttest local cluster with a real topo server #9176

    Feature Request

    Query Serving

    • Add support for SET statements in OLAP mode #9253
    • Multi-column vindex support in select statements #9326
    • Multi-column vindex support in update and delete queries #9467
    • Multicolumn partial vindex selection #9390
    • Subsharding vindex support #9428
    • Non-unique vindex routing in update and delete query #9554

    VTAdmin

    • [vtadmin] Add create+delete shards functionality #9012
    • Add experimental whoami api endpoint #9233
    • [vtadmin] Add ping feature, Dropdown and Dialog components #9318
    • [VTAdmin] Dynamic Cluster support #9544

    VTorc

    • Merge InitShardPrimary functionality into PlannedReparentShard #9276

    Internal Cleanup

    Backup and Restore

    • Inclusive naming: use new RPCs in vtctl/vtctld/wrangler etc. #9181

    Cluster management

    • reparent_journal: add backwards compatible alter statement #9439
    • [vtctl] command deprecations #8967
    • [wrangler] Cleanup duplicate wrangler methods #9015
    • Migrate k8s topo CRD to v1 api #9045
    • [schematools] Move more functions out of wrangler to package importable by grpcvtctldserver #9123
    • [vtctld] Migrate ReloadSchema* rpcs #8832
    • [vtctldserver] migrate ExecuteHook #9024
    • [vtctldclient-codegen] Add support in codegen for streaming RPCs #9064

    General

    • Delete unused go/cmd/automation_{client,server} binaries #9234
    • Inclusive naming: replace master with primary #9427,#9430,#9438,#9454,#9103,#9182
    • naming: delete old code that was needed for version compatibility #9516
    • MySQL: Pass mysql.Conn through {Hash,PlainText,Caching} storage interfaces #9264

    Java

    • build(deps): bump log4j-api and log4j-core from 2.13.3 to 2.15.0 in /java #9348,#9349

    Observability

    • Changed master to primary everywhere #9506

    Query Serving

    • Deprecate and ignore gateway_implementation flag to vtgate #9482
    • Deprecate vtctl command IgnoreHealthError #9594

    VReplication

    • Fixed legacy vreplication data that may be using master tablet types #9497

    VTAdmin

    • [vtadmin-web] Upgrade to node v16.13.0 LTS #9136

    VTorc

    • Fix linting issues in Vtorc #9449
    • Pass Semi sync information from durability policy and use it to log discrepancies #9533

    vtctl

    • Inclusive naming: removing vtctl flags and commands with master in them that were deprecated in 12.0 #9179
    • vtctl: delete deprecated blacklisted_tables flag #9406

    web UI

    • Migrate jQuery (pt. 1) #9239
    • Migrate jQuery (pt. 2/3) #9256

    Performance

    Query Serving

    • vttablet: Better use of vtprotobuf memory pool #9365

    Testing

    Query Serving

    • Fuzzing: Add more fuzzers #9249

    The release includes 2020 commits (excluding merges)

    Thanks to all our contributors: @AdamKorcz, @FancyFane, @GuptaManan100, @Phanatic, @Thirumalai-Shaktivel, @ajm188, @aquarapid, @arthurschreiber, @arvind-murty, @askdba, @carsonoid, @chapsuk, @choo-stripe, @dasl-, @dbussink, @dctrwatson, @deepthi, @dependabot[bot], @derekperkins, @doeg, @frouioui, @hallaroo, @harshit-gangal, @hkdsun, @king-11, @mattlord, @mattrobenolt, @mvh-stripe, @notfelineit, @oscerd, @pH14, @ritwizsinha, @rohit-nayak-ps, @shichao-an, @shlomi-noach, @systay, @utk9, @vmg, @y5w, @zhongr3n

    Source code(tar.gz)
    Source code(zip)
    vitess-13.0.0-bc4a960.tar.gz(463.06 MB)
    vitess-13.0.0-bc4a960.x86_64.rpm(461.37 MB)
    vitess_13.0.0-bc4a960_amd64.deb(463.07 MB)
  • v13.0.0-rc1(Feb 1, 2022)

    Release of Vitess v13.0.0-rc1

    Major Changes

    Vitess now has native support for MySQL collations

    When using the gen4 planner, Vitess is now capable of performing collation-aware string comparisons in the vtgates. This improves the performance and reliability of several query plans that were previously relying on a debug-only SQL API in MySQL to perform these comparisons remotely. It also enables new query plans that were previously not possible.

    A full list of the supported collations can be found in the Vitess documentation.

    The native Evaluation engine in vtgate has been greatly improved

    The SQL evaluation engine that runs inside the vtgates has been rewritten mostly from scratch to more closely match MySQL's behavior. This allows Vitess to execute more parts of the query plans locally, and increases the complexity and semantics of the SQL expressions that can be used to perform cross-shard queries.

    vttablet -use_super_read_only flag now defaults to true

    The default value used to be false. What this means is that during a failover, we will set super_read_only on database flavors that support them (MySQL 5.7+ and Percona 5.7+). In addition, all Vitess-managed databases will be started with super-read-only in the cnf file. It is expected that this change is safe and backwards-compatible. Anyone who is relying on the current behavior should pass -use_super_read_only=false on the vttablet command line, and make sure they are using a custom my.cnf instead of the one provided as the default by Vitess.

    vtgate -buffer_implementation now defaults to keyspace_events

    The default value used to be healthcheck. The new keyspace_events implementation has been tested in production with good results and shows more consistent buffering behavior during PlannedReparentShard operations. The keyspace_events implementation utilizes heuristics to detect additional cluster states where buffering is safe to perform, including cases where the primary may be down. If there is a need to revert back to the previous buffer implementation, ensure buffering is enabled in vtgate and pass the flag -buffer_implementation=healthcheck.

    ddl_strategy: -postpone-completion flag

    ddl_strategy (either @@ddl_strategy in VtGate or -ddl_strategy in vtctlclient ApplySchema) supports the flag -postpone-completion

    This flag indicates that the migration should not auto-complete. This applies for:

    • any CREATE TABLE
    • any DROP TABLE
    • ALTER table in online strategy
    • ALTER table in gh-ost strategy

    Note that this flag is not supported for pt-osc strategy.

    Behavior of migrations with this flag:

    • an ALTER table begins, runs, but does not cut-over.
    • CREATE or DROP migrations are silently not even scheduled

    alter vitess_migration ... cleanup

    A new query is supported:

    alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' cleanup
    

    This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As a reminder, once a migration's artifacts are cleaned up, the migration is no longer revertible.

    alter vitess_migration ... complete

    A new query is supported:

    alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete
    

    This command indicates that a migration executed with -postpone-completion is good to complete. Behavior:

    • For running ALTERs (online and gh-ost) which are ready to cut-over: cut-over imminently (though not immediately
      • cut-over depends on polling interval, replication lag, etc)
    • For running ALTERs (online and gh-ost) which are only partly through the migration: they will cut-over automatically when they complete their work, as if -postpone-completion wasn't indicated
    • For queued CREATE and DROP migrations: "unblock" them from being scheduled. They'll be scheduled at the scheduler' s discretion. there is no guarantee that they will be scheduled to run immediately.

    vtctl/vtctlclient ApplySchema: ALTER VITESS_MIGRATION

    vtctl ApplySchema now supports ALTER VITESS_MIGRATION ... statements. Example:

    $ vtctl ApplySchema -skip_preflight -sql "alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete" commerce
    

    vtctl/vtctlclient ApplySchema: allow zero in date

    vtctl/vtctlclient ApplySchema now respects -allow-zero-in-date for direct strategy. For example, the following statement is now accepted:

    vtctlclient ApplySchema -skip_preflight -ddl_strategy='direct -allow-zero-in-date' -sql "create table if not exists t2(id int primary key, dt datetime default '0000-00-00 00:00:00')" commerce
    

    vtctl/vtctlclient ApplySchema -uuid_list

    vtctlient ApplySchema now support a new optional -uuid_list flag. It is possible for the user to explicitly specify the UUIDs for given migration(s). UUIDs must be in a specific format. If given, number of UUIDs must match the number of DDL statements. Example:

    vtctlclient OnlineDDL ApplySchema -sql "drop table t1, drop table t2" -uuid_list "d08f0000_51c9_11ec_9cf2_0a43f95f28a3,d08f0001_51c9_11ec_9cf2_0a43f95f28a3" commerce
    

    Vitess will assign each migration with given UUID in order of appearance. It is the user's responsibility to ensure given UUIDs are globally unique. If the user submits a migration with an already existing UUID, that migration never gets scheduled nor executed.

    vtctl/vtctlclient ApplySchema -migration_context

    -migration_context flag is synonymous to -request_context. Either will work. We will encourage use of -migration_context as it is more consistent with output of SHOW VITESS_MIGRATIONS ... which includes the migration_context column.

    vtctl/vtctlclient OnlineDDL ... complete

    Complementing the alter vitess_migration ... complete query, a migration can also be completed via vtctl or vtctlclient:

    vtctlclient OnlineDDL <keyspace> complete <uuid>
    

    For example:

    vtctlclient OnlineDDL commerce complete d08ffe6b_51c9_11ec_9cf2_0a43f95f28a3
    

    vtctl/vtctlclient OnlineDDL -json

    The command now accepts an optional -json flag. With this flag, the output is a valid JSON listing all columns and rows.

    vtadmin-web updated to node v16.13.0 (LTS)

    Building vtadmin-web now requires node >= v16.13.0 (LTS). Upgrade notes are given in https://github.com/vitessio/vitess/pull/9136.

    PlannedReparentShard for cluster initialization

    For setting up the cluster and electing a primary for the first time, PlannedReparentShard should be used instead of InitShardPrimary.

    InitShardPrimary is a forceful command and copies over the executed gtid set from the new primary to all the other replicas. So, if the user isn't careful, it can lead to some replicas not being setup correctly and lead to errors in replication and recovery later. PlannedReparentShard is a safer alternative and does not change the executed gtid set on the replicas forcefully. It is the preferred alternate to initialize the cluster.

    If using a custom init_db.sql that omits SET sql_log_bin = 0, then InitShardPrimary should still be used instead of PlannedReparentShard.

    Durability Policy flag

    A new flag has been added to vtctl, vtctld and vtworker binaries which allows the users to set the durability policies.

    If semi-sync is not being used then -durability_policy should be set to none. This is also the default option.

    If semi-sync is being used then -durability_policy should be set to semi_sync and -enable_semi_sync should be set in vttablets.

    Incompatible Changes

    Error message change when vttablet row limit exceeded:

    • In previous Vitess versions, if the vttablet row limit (-queryserver-config-max-result-size) was exceeded, an error like:
      ERROR 10001 (HY000): target: unsharded.0.master: vttablet: rpc error: code = ResourceExhausted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      would be reported to the client.
      

      To avoid confusion, the error mapping has been changed to report the error as similar to:

      ERROR 10001 (HY000): target: unsharded.0.primary: vttablet: rpc error: code = Aborted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      instead
      
    • Because of this error code change, the vttablet metric: vttablet_errors{error_code="ABORTED"} will be incremented upon this type of error, instead of the previous metric: vttablet_errors{error_code="RESOURCE_EXHAUSTED"}
    • In addition, the vttablet error message logged is now different. Previously, a (misleading; due to the PoolFull) error was logged:
      E0112 09:48:25.420641  278125 tabletserver.go:1368] PoolFull: Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      

      Post-change, a more accurate warning is logged instead:

      W0112 09:38:59.169264   35943 tabletserver.go:1503] Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      

    Column types for textual queries now match MySQL's behavior

    The column types for certain queries performed on a vtgate (most notably, those that SELECT system variables) have been changed to match the types that would be returned if querying a MySQL instance directly: textual fields that were previously returned as VARBINARY will now appear as VARCHAR.

    This change should not have an impact on MySQL clients/connectors for statically typed programming languages, as these clients coerce the returned rows into whatever types the user has requested, but clients for dynamic programming languages may now start returning as "string" values that were previously returned as "bytes".

    Deprecations

    vtgate -gateway_implementation flag is deprecated (and ignored)

    Support for discoverygateway is being dropped. tabletgateway is now the only supported implementation. Scripts using this flag should be updated to remove the flag as it will be deleted in the next release.


    Changelog

    Bug fixes

    Backup and Restore

    • vtctl: Check ShardReplicationStatuses error and make sure they are well returned #8966

    Cluster management

    • Fix vtctlclient DeleteCellInfo(force=true) command with downed local topo #9081
    • Fix race condition when rapidly starting/stopping health check #9152
    • Topo write error in change tablet type #9157
    • Default build info tablet tags to off #9187
    • Replication manager should not start health ticks after calling Promote Replica #9274
    • Fix error checking in ERS #9330
    • Fix for context cancellation in ERS #9503

    General

    • Avoid some pollution of CLI argument namespace across vtgate and vttablet #8931
    • Update to the latest Tengo version #9110

    Orchestrator

    • Set orc maintenance mode on tablet that is being promoted by PRS #8859

    Query Serving

    • Fix for schema tracker not recognizing RENAME for schema reloading #8963
    • Handle sql_mode differently for new value change #9014
    • Fix savepoint support with reserved connections #9030
    • Use a DBA user pool for the heartbeat writer #9033
    • Handle single column vindex correctly with multi-columns in Gen4 #9060
    • Remove tablet healthcheck cache record on error #9106
    • Fix request buffering while reparenting #9112
    • Ensure that the hex query predicates are normalized for planner cache #9118
    • Fix table/sizes query, to allow vttablet schema engine to see partitioned tables #9188
    • Remove keyspace from the query before sending it to the shards #9190
    • Make sure to copy bindvars when using them concurrently #9191
    • Fix owned vindex query to keep alias in table expression #9244
    • Use decoded hex string when calculating the keyspace ID #9277
    • Fix show statement in prepare call #9280
    • Avoid parenthesis around default null in BLOB, TEXT, GEOMETRY, and JSON column #9301
    • Allow unowned lookup_unique vindex columns to be null #9302
    • Estimate replica lag when the number of seconds behind from mysqld is unknown #9308
    • Clear existing keyspace schema in vtgate before [re]loading it #9437
    • Fix AST copy issue in the Gen4Fallback planner #9487
    • Fix a bug in the Gen4 planner for join query pushing output column on the left-hand side #9521

    TabletManager

    • Prevent Race Conditions Between Tablet Deletes and Updates #9237

    VReplication

    • Fix boolean parameter order in DropSources call for v2 flows #9175
    • VReplication Workflow Create: Check for Copying state while waiting for streams to start #9206
    • MoveTables: update vschema while moving tables with autoincrement from sharded to unsharded #9288
    • Schema Engine now accountable for Online DDL swapped tables #9324
    • Take MySQL Column Type Into Account in VStreamer #9331
    • Allow source columns to be specified multiple times in a vreplication rule filter #9335
    • Fixed case-sensitive TabletType in SwitchTraffic #9440
    • Use LOCK TABLES during SwitchWrites to prevent lost writes #9481
    • VReplication Workflow: manage SQL_MODE as MySQL does in mysqldump #9505
    • Routing Rules: Improved escaping of reserved and invalid names for tables and keyspaces #9522
    • Ignore internal tables in MoveTables and Reshard Workflows #8992,#9578

    VTCombo

    • vtcombo: CREATE DATABASE creates shards as specified by the topology #9130

    CI/Build

    Build/CI

    • Remove Percona5.6 Unit Tests for Vitess 13.0 #9067
    • Addition of upgrade downgrade testing to the test suite #9300, #9473, #9501
    • Re-enable default Make behavior of failing target if any command fails #9319
    • ubuntu-latest now uses MySQL 8.0.26, let us override it with latest 8.0.x #9368
    • Add vtadmin binary to release and Docker images #9405

    VTAdmin

    • Enforce node + npm versions with .npmrc #9336

    vtexplain

    • Correct handling of column types for queries involving multiple tables #9260

    Dependabot

    Build/CI

    • Upgrade go-proxyproto to v0.6.1 to fix vulnerabilities #9425

    Java

    • Bump log4j-core from 2.15.0 to 2.16.0 in /java #9387
    • Bump log4j-api from 2.16.0 to 2.17.1 in /java #9423
    • Updates all major Java dependencies to latest #9599

    VTAdmin

    • Bump tmpl from 1.0.4 to 1.0.5 in /web/vtadmin #9278

    Enhancement

    Authn/z

    • Add support for TLS certification revocation list (CRL) files #8807

    Backup and Restore

    • Set super_read_only off during restore (achieves same as PR #8929) #9240

    Cluster management

    • Support -allow-zero-in-date in ApplySchema 'direct' strategy #9273
    • Remove legacy OnlineDDL code deprecated in earlier version #8971
    • Add version-info to tablet tags in the topo #8973
    • Add default collation to VTGate and VTTablet #9097
    • Use relay log position in PRS while choosing the new primary #9270
    • vtctl OnlineDDL supports 'complete' command #9298
    • vtctl ApplySchema accepts 'ALTER VITESS_MIGRATION...' statements #9303
    • Make the heartbeat writer use 2 pools #9320
    • New vtctl ApplySchema flag -uuid_list #9325
    • Fix ERS to be used for initialization as well #9511
    • Fix ERS to work when the primary candidate's replication is stopped #9512

    Examples

    • Update operator.yaml to v2.5.0 #9016

    General

    • vitessdriver: add support for DistributedTx #9451

    Observability

    • Modify terse errors to also redact errors in logs #9094

    Query Serving

    • Initial implementation of partitions (Create statement) #8691
    • Add parsing for common table expressions using WITH clause #8918
    • Support SQL SELECT LIMIT #8944
    • Support for union in subquery in Gen4 #8948
    • Support for ordering using derived table columns in Gen4 #8961
    • Add planner-version flag to vtexplain #8979
    • Rewrite NOT expressions for cleaner plans #8987
    • Minimize logging of errors when loading keyspace with disabled schema tracking #8989
    • Support for ordering on vindex function and straight_join queries to be planned by ignoring the hint as warning in Gen4 #8990
    • Improve internal subquery dependencies update in Gen4 #8998
    • Added tokenizer support for NCHAR_STRING #9006
    • Query timeout in comment directive in Gen4 #9008
    • Added brackets in default value if column is of type TEXT, BLOB, GEOMETRY or JSON #9011
    • Addition of the filter primitive and planning in Gen4 #9017
    • Initial integration of the collation module in Gen4 #9018
    • Support extract function in the SQL parser #9029
    • Add collation name to the type definition in Gen4 #9038
    • Support semi-join in Gen4 #9039
    • Changed Collations and Expression grammar #9075
    • Support transactions with OLAP workload #9115
    • Use of advisory lock in prepare queries #9129
    • Add hash join primitive and planning in Gen4 #9140
    • Respect --allow-zero-in-date in CREATE and in declarative migrations #9142
    • Improve merging of DBA queries in Gen4 #9183
    • Evaluate REVERTibility of a migration in OnlineDDL #9232
    • Improve group by queries planning when they use a sharding key #9243
    • Make sure VTGate and VTTablet collations are matching #9248
    • Support filtering on derived union system table with star projection #9263
    • MySQL: Better handling of query failures within a transaction to match the MySQL behavior #9269
    • Add Tuples and InOp implementation to EvalEngine #9281
    • Use evalengine for Vindex lookups in route primitive #9291
    • Adds parsing support for string introducers #9309
    • Multi-column vindex support for DML #9338
    • Add list (IN()) support to vindex functions #9426
    • Fix parsing of table names that start with a number #9456
    • Make the MaxWaiters setting configurable for the OLAP pool #9484
    • Add support for health service in gRPC server #9528
    • Filter after left join #9531
    • Add Information_Schema to evalengine #9536
    • Faster and safer unsharded query planning #9542
    • Add support for PLANNER SQL comment #9545
    • MySQL: use UTF8MB4 consistently as the connection charset #9558
    • Do not auto retry after "server lost" errors (errno 2013) in vttablet #9009
    • Online DDL: migration with same context and SQL now considered duplicate and silently marked as complete #9107
    • Support ALTER VITESS_MIGRATION ... CLEANUP statement #9160
    • OnlineDDL: implementing -postpone-completion, ALTER VITESS_MIGRATION ... COMPLETE #9171
    • OnlineDDL: support concurrent REVERT migrations #9192
    • VTGate: change default query buffer implementation (used to avoid client errors during state transitions such as PlannedReparentShard) #9359 #9360
    • Fix misleading error mapping for row limit error #9448

    TabletManager

    • Have vttablet use super_read_only by default when a tablet becomes a replica #9312
    • MySQL: Add ConnectionReady callback to Handler interface #9496
    • MySQL: Add UnimplementedHandler struct #9526

    VReplication

    • Support VStream with keyspaces_to_watch #8988
    • VEvent: Add Keyspace/Shard properties to proto #9386
    • Initial refactoring of the 'vstream * from' functionality in vtgate #9392
    • Add flag to keep routing rules in v2 vrepl workflows #9441
    • SwitchTraffic: check vreplication lag before switching #9538

    VTAdmin

    • [vtadmin] viper config support #9154
    • [VTAdmin] Update react-scripts, postcss #9493

    VTorc

    • Have a common EmergencyReparentShard (ERS) implementation for both VtOrc and Vtctl #8492
    • Use PlannedReparentShard (PRS) in VtOrc while doing Graceful Primary Takeover #9258
    • Make PRS use Durability Rules #9259
    • Use PRS in Vtorc for electing new primary #9409

    vtctl

    • ListAllTablets: improve efficiency within vitess (fewer topo calls) and for caller (keyspace and tablet type filters) #9560
    • vtctl OnlineDDL show: export all columns, support -json #9568
    • vtctl ApplySchema: introduce -migration_context flag, synonym to -request_context #9572

    vttestserver

    • Add option to run vtcombo and vttest local cluster with a real topo server #9176

    Feature Request

    Query Serving

    • Add support for SET statements in OLAP mode #9253
    • Multi-column vindex support in select statements #9326
    • Multi-column vindex support in update and delete queries #9467
    • Multicolumn partial vindex selection #9390
    • Subsharding vindex support #9428
    • Non-unique vindex routing in update and delete query #9554

    VTAdmin

    • [vtadmin] Add create+delete shards functionality #9012
    • Add experimental whoami api endpoint #9233
    • [vtadmin] Add ping feature, Dropdown and Dialog components #9318
    • [VTAdmin] Dynamic Cluster support #9544

    VTorc

    • Merge InitShardPrimary functionality into PlannedReparentShard #9276

    Internal Cleanup

    Backup and Restore

    • Inclusive naming: use new RPCs in vtctl/vtctld/wrangler etc. #9181

    Cluster management

    • reparent_journal: add backwards compatible alter statement #9439
    • [vtctl] command deprecations #8967
    • [wrangler] Cleanup duplicate wrangler methods #9015
    • Migrate k8s topo CRD to v1 api #9045
    • [schematools] Move more functions out of wrangler to package importable by grpcvtctldserver #9123
    • [vtctld] Migrate ReloadSchema* rpcs #8832
    • [vtctldserver] migrate ExecuteHook #9024
    • [vtctldclient-codegen] Add support in codegen for streaming RPCs #9064

    General

    • Delete unused go/cmd/automation_{client,server} binaries #9234
    • Inclusive naming: replace master with primary #9427,#9430,#9438,#9454,#9103,#9182
    • naming: delete old code that was needed for version compatibility #9516
    • MySQL: Pass mysql.Conn through {Hash,PlainText,Caching} storage interfaces #9264

    Java

    • build(deps): bump log4j-api and log4j-core from 2.13.3 to 2.15.0 in /java #9348,#9349

    Observability

    • Changed master to primary everywhere #9506

    Query Serving

    • Deprecate and ignore gateway_implementation flag to vtgate #9482
    • Deprecate vtctl command IgnoreHealthError #9594

    VReplication

    • Fixed legacy vreplication data that may be using master tablet types #9497

    VTAdmin

    • [vtadmin-web] Upgrade to node v16.13.0 LTS #9136

    VTorc

    • Fix linting issues in Vtorc #9449
    • Pass Semi sync information from durability policy and use it to log discrepancies #9533

    vtctl

    • Inclusive naming: removing vtctl flags and commands with master in them that were deprecated in 12.0 #9179
    • vtctl: delete deprecated blacklisted_tables flag #9406

    web UI

    • Migrate jQuery (pt. 1) #9239
    • Migrate jQuery (pt. 2/3) #9256

    Performance

    Query Serving

    • vttablet: Better use of vtprotobuf memory pool #9365

    Testing

    Query Serving

    • Fuzzing: Add more fuzzers #9249

    The release includes 1963 commits (excluding merges)

    Thanks to all our contributors: @AdamKorcz, @FancyFane, @GuptaManan100, @Phanatic, @Thirumalai-Shaktivel, @ajm188, @aquarapid, @arthurschreiber, @arvind-murty, @askdba, @carsonoid, @chapsuk, @choo-stripe, @dasl-, @dbussink, @dctrwatson, @deepthi, @dependabot[bot], @derekperkins, @doeg, @frouioui, @hallaroo, @harshit-gangal, @hkdsun, @king-11, @mattlord, @mattrobenolt, @mvh-stripe, @notfelineit, @oscerd, @pH14, @ritwizsinha, @rohit-nayak-ps, @shichao-an, @shlomi-noach, @systay, @vmg, @y5w, @zhongr3n

    Source code(tar.gz)
    Source code(zip)
    vitess-13.0.0-rc1-3fa0fba.tar.gz(463.01 MB)
    vitess-13.0.0_rc1-3fa0fba.x86_64.rpm(461.35 MB)
    vitess_13.0.0-rc1-3fa0fba_amd64.deb(463.01 MB)
  • v12.0.3(Jan 10, 2022)

    Release of Vitess v12.0.3

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44832) (#9465).


    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-api from 2.16.0 to 2.17.1 in /java #9465

    The release includes 4 commits (excluding merges)

    Thanks to all our contributors: @dbussink, @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.3-cbc6ab6.tar.gz(400.48 MB)
    vitess-12.0.3-cbc6ab6.x86_64.rpm(398.75 MB)
    vitess_12.0.3-cbc6ab6_amd64.deb(400.49 MB)
  • v11.0.4(Jan 10, 2022)

    Release of Vitess v11.0.4

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44832) (#9464).

    Known Issues

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-api from 2.16.0 to 2.17.1 in /java #9464

    The release includes 6 commits (excluding merges)

    Thanks to all our contributors: @dbussink, @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.4-cb142ee.tar.gz(397.17 MB)
    vitess-11.0.4-cb142ee.x86_64.rpm(394.91 MB)
    vitess_11.0.4-cb142ee_amd64.deb(397.17 MB)
  • v10.0.5(Jan 10, 2022)

    Release of Vitess v10.0.5

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44832) (#9463).

    Known Issues

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-api from 2.16.0 to 2.17.1 in /java #9463

    The release includes 7 commits (excluding merges)

    Thanks to all our contributors: @dbussink, @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.5-dec816c.tar.gz(387.07 MB)
    vitess-10.0.5-dec816c.x86_64.rpm(384.80 MB)
    vitess_10.0.5-dec816c_amd64.deb(387.08 MB)
  • v12.0.2(Dec 17, 2021)

    Release of Vitess v12.0.2

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-45046) (#9396).

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs (CVE-2021-45046 and CVE-2021-44832) followed. These have been fixed in release 2.17.1. This release of Vitess, v12.0.2, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v12.0.3 instead, to benefit from the vulnerability patches.

    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-core from 2.15.0 to 2.16.0 in /java #9396

    The release includes 2 commits (excluding merges)

    Thanks to all our contributors: @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.2-d620dab.tar.gz(400.48 MB)
    vitess-12.0.2-d620dab.x86_64.rpm(398.75 MB)
    vitess_12.0.2-d620dab_amd64.deb(400.49 MB)
  • v11.0.3(Dec 17, 2021)

    Release of Vitess v11.0.3

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-45046) (#9395).

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v11.0.3, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v11.0.4 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.


    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-core from 2.15.0 to 2.16.0 in /java #9395

    Documentation

    Examples

    • change operator example to use v11.0.3 docker images #9403

    The release includes 3 commits (excluding merges) Thanks to all our contributors: @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.3-2412024.tar.gz(397.20 MB)
    vitess-11.0.3-2412024.x86_64.rpm(394.93 MB)
    vitess_11.0.3-2412024_amd64.deb(397.20 MB)
  • v10.0.4(Dec 17, 2021)

    Release of Vitess v10.0.4

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-45046) (#9394).

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v10.0.4, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v10.0.5 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.


    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-core from 2.15.0 to 2.16.0 in /java #9394

    Documentation

    Examples

    • change operator example to use v10.0.4 docker images #9402

    The release includes 2 commits (excluding merges) Thanks to all our contributors: @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.4-23c5cf9.tar.gz(387.07 MB)
    vitess-10.0.4-23c5cf9.x86_64.rpm(384.80 MB)
    vitess_10.0.4-23c5cf9_amd64.deb(387.08 MB)
  • v12.0.1(Dec 14, 2021)

    Release of Vitess v12.0.1

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44228) (#9357), along with a few bug fixes.

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v12.0.1, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v12.0.3 instead, to benefit from the vulnerability patches.

    Changelog

    Bug fixes

    Query Serving

    • Ensure that hex query predicates are normalized for planner cache #9145
    • Gen4: Fail cross-shard join query with aggregation and grouping #9167
    • Make sure to copy bindvars when using them concurrently #9246
    • Remove keyspace from query before sending it on #9247
    • Use decoded hex string when calculating the keyspace ID #9293

    VReplication

    • Fix boolean parameter order in DropSources call for v2 flows #9178
    • Take MySQL Column Type Into Account in VStreamer #9355

    Cluster management

    • Restoring 'vtctl VExec' command #9227
      • This change restores vtctl VExec functionality. It was removed based on the assumption the only uses for this command were for Online DDL command. This was wrong, and VExec is also used as a wrapper around VReplication.

    CI/Build

    Build/CI

    • CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x #9373

    Internal Cleanup

    Java

    • build(deps): bump log4j-api from 2.13.3 to 2.15.0 in /java #9357

    The release includes 21 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @askdba, @deepthi, @dependabot[bot], @frouioui, @hallaroo, @harshit-gangal, @mattlord, @rohit-nayak-ps, @shlomi-noach, @systay, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.1-558a5eb.tar.gz(400.46 MB)
    vitess-12.0.1-558a5eb.x86_64.rpm(398.75 MB)
    vitess_12.0.1-558a5eb_amd64.deb(400.47 MB)
  • v11.0.2(Dec 14, 2021)

    Release of Vitess v11.0.2

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44228) (#9364), along with a few bug fixes.

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v11.0.2, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v11.0.4 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.


    Changelog

    Bug fixes

    VReplication

    • Fix how we identify MySQL generated columns #8796

    CI/Build

    Build/CI

    • CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x #9374

    Internal Cleanup

    Java

    • build(deps): bump log4j-api from 2.13.3 to 2.15.0 in /java #9364

    The release includes 7 commits (excluding merges)

    Thanks to all our contributors: @askdba, @deepthi, @systay, @tokikanno

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.2-4f80587.tar.gz(397.20 MB)
    vitess-11.0.2-4f80587.x86_64.rpm(394.93 MB)
    vitess_11.0.2-4f80587_amd64.deb(397.20 MB)
  • v10.0.3(Dec 14, 2021)

    Release of Vitess v10.0.3

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44228) (#9363).

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v10.0.3, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v10.0.5 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.


    Changelog

    CI/Build

    Build/CI

    CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x #9375

    Internal Cleanup

    Java

    • build(deps): bump log4j-api from 2.13.3 to 2.15.0 in /java #9363

    The release includes 5 commits (excluding merges)

    Thanks to all our contributors: @deepthi, @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.3-a267978.tar.gz(387.07 MB)
    vitess-10.0.3-a267978.x86_64.rpm(384.80 MB)
    vitess_10.0.3-a267978_amd64.deb(387.08 MB)
  • v12.0.0(Oct 26, 2021)

    Release of Vitess v12.0.0

    Major Changes

    This release includes the following major changes or new features.

    Inclusive Naming

    A number of CLI commands and vttablet RPCs have been deprecated as we move from master to primary. All functionality is backwards compatible except as noted under Incompatible Changes. Deprecated commands and flags will be removed in the next release (13.0). Deprecated vttablet RPCs will be removed in the subsequent release (14.0).

    Gen4 Planner

    The newest version of the query planner, Gen4, becomes an experimental feature as part of this release. While Gen4 has been under development for a few release cycles, we have now reached parity with its predecessor, v3.

    To use Gen4, VTGate's -planner_version flag needs to be set to Gen4Fallback.

    Query Buffering during failovers

    In order to support buffering during resharding cutovers in addition to primary failovers, a new implementation of query buffering has been added. This is considered experimental. To enable it the flag buffer_implementation should be set to keyspace_events. The existing implementation (flag value healthcheck) will be deprecated in a future release.

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v12.0.0, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v12.0.3 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174. This issue is fixed in release >= v12.0.1.

    Incompatible Changes

    CLI command output

    Wherever CLI commands produced master or MASTER for tablet type, they now produce primary or PRIMARY. Scripts and tools that depend on parsing command output will need to change.

    Example:

    $ vtctlclient -server localhost:15999 ListAllTablets 
    zone1-0000000100 sourcekeyspace 0 primary 192.168.0.134:15100 192.168.0.134:17100 [] 2021-09-24T01:12:00Z
    zone1-0000000101 sourcekeyspace 0 rdonly 192.168.0.134:15101 192.168.0.134:17101 [] <null>
    zone1-0000000102 sourcekeyspace 0 replica 192.168.0.134:15102 192.168.0.134:17102 [] <null>
    zone1-0000000103 sourcekeyspace 0 rdonly 192.168.0.134:15103 192.168.0.134:17103 [] <null>
    

    Default Behaviour Changes

    Enabling reserved connections

    Use of reserved connections is controlled by the vtgate flag -enable_system_settings. This flag has in the past defaulted to false (or OFF) in release versions (i.e. x.0 and x.0.y) of Vitess, and to true (or ON) in development versions.

    From Vitess 12.0.0 onwards, it defaults to true in all release and development versions. You can read more about this here. Hence you should specify this flag explicitly, so that you are sure whether it is enabled or not, regardless of which Vitess release/build/version you are running.

    If you have reserved connections disabled, you will get the old Vitess behavior: where most system settings (e.g. sql_mode) are just silently ignored by Vitess. In situations where you know your backend MySQL defaults are acceptable, this may be the correct setting to ensure the best possible performance of the vttablet <-> MySQL connection pools. This comes down to a trade-off between compatibility and performance/scalability. You should also review this section of the documentation when deciding whether or not to enable reserved connections.

    Deprecations

    CLI commands

    VExec is deprecated and removed. All Online DDL commands should be run through OnlineDDL.

    OnlineDDL revert is deprecated. Use REVERT VITESS_MIGRATION '...' SQL command either via ApplySchema or via vtgate.

    InitShardMaster is deprecated, use InitShardPrimary instead.

    SetShardIsMasterServing is deprecated, use SetShardIsPrimaryServing instead.

    Various command flags have been deprecated and new variants provided.

    • DeleteTablet flag allow_master replaced with allow_primary
    • PlannedReparentShard flag avoid_master replaced with avoid_tablet
    • PlannedReparentShard flag new_master replaced with new_primary
    • BackupShard flag allow_master replaced with allow_primary
    • Backup flag allow_master replaced with allow_primary
    • EmergencyReparentShard flag new_master replaced with new_primary
    • ReloadSchemeShard flag include_master replaced with include_primary
    • ReloadSchemaKeyspace flag include_master replaced with include_primary
    • ValidateSchemaKeyspace flag skip-no-master replaced with skip-no-primary

    Minor Changes

    Query Serving

    • Add SHOW VITESS_REPLICATION_STATUS Query Support #8900
      • The new SHOW VITESS_REPLICATION_STATUS command/query shows the MySQL replica/replication (not vreplication) health for the vitess deployment.

    Governance

    • Added @GuptaManan100 to maintainers #8833

    Changelog

    Bug fixes

    Build/CI

    • Update golang/x dependencies #8688
    • tests: use AtomicInt32 instead of int to fix races #8696

    Cluster management

    • Use consul lock properly #8310
    • Trivial: mysqlctl reinit_config -h was not showing help #8369
    • Port #8422 to main branch #8745
    • [vtctl] Add missing call to flag.Parse in commandGetRoutingRules #8795
    • Fix for padding in OrderAndCheckPartitions #8873
    • tabletmanager: correct implementations of deprecated RPCs for backwards compatibility #9059

    General

    • Avoid some pollution of CLI argument namespace across vtgate and vttablet #9026

    Observability

    • schema engine: reset table size stats when a table is dropped #8634

    Query Serving

    • Fix for function calls in DEFAULT value of CREATE TABLE statement in main #8477
    • Fixing multiple issues related to onlineddl/lifecycle #8500
    • boolean values should not be parenthesised in default clause #8502
    • mysql/server: abort connection if secure transport is required #8504
    • Clear only the current shard's schemacopy rows during schema reload #8519
    • go/mysql/auth: fix error message for wrong auth method. #8525
    • Fix handshake protocol with MySQL client #8537
    • Copy parameter data from byte buffer in COM_STMT_SEND_LONG_DATA #8562
    • Negative float default #8587
    • [vtexplain] prevent panic on show tables/show full tables #8601
    • OnlineDDL: better scheduling/cancellation logic #8603
    • Fix parsing of FIRST clause in ALTER TABLE #8614
    • onlineddl Executor: build schema with DBA user #8624
    • Handle subquery merging with references correctly #8662
    • Fixing error on deletes from owning table for a lookup being populated by CreateLookupVindex after a SwitchWrites #8701
    • Fixing a panic in vtgate with OLAP mode #8722
    • default to primary tablet if not set in VStream api #8755
    • Fix bug in parsing of version comments #8770
    • Proper merge of the SysTableTableName fields when joining two routes #8771
    • fix parsing of MySQL Server Version flag #8791
    • Fix typo in pool config syntax. Fixes #8819 #8820
    • VtGate schema tracker to use provided user #8857
    • fix merging dba query as a subquery both in v3 and gen4 #8871
    • sql_select_limit support #9027
    • [12.0] Handle sql_mode setting to compare modes in any order #9041
    • [12.0] Fix for schema tracker not recognizing RENAME for schema reloading #9043
    • [12.0] Fix savepoint support with reserved connections #9047
    • [12.0] Gen4: Handle single column vindex correctly with multi column #9061

    VReplication

    • MoveTables: don't create unnecessary streams on the target for non-intersecting sources and targets #8090
    • VDiff: Add BIT datatype to list of byte comparable types #8401
    • Fix vreplication error metric #8483
    • Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
    • Fix VReplication logging to file and db #8521
    • Fix passing the wrong cell/cells variable to CreateLookupVindex #8590
    • Refresh SrvVSchema after an ExternalizeVindex: was missing #8670
    • Fixing missing argument in function call #8692
    • Fix how we identify MySQL generated columns #8763

    VTAdmin

    • [vtadmin-web] Set Backup status indicators to the correct colours #8410
    • [vtadmin] Fix bad copy-paste in pool config flag parsing #8518

    vtctl

    • Print actual current datetime for vtctl dry run commands #8778

    vttestserver

    • Remove lingering mysqld sock files on vttestserver container startup #8463
    • Fixed filename for the configuration file in Vttestserver #8809

    CI/Build

    Build/CI

    • Update release process #7759
    • [ci] Add errcheck to golangci-lint #7996
    • gomod: do not replace GRPC #8416
    • consolidation: Fix flaky test #8417
    • Small build improvements #8418
    • proto: upgrade vtprotobuf version #8454
    • Automatically use the latest tag of Vitess for cluster upgrade E2E test #8471
    • Addition of known issues to release notes #8482
    • Fix Cluster 14 flakiness #8494
    • add question on backporting back to PR template #8541
    • Enable GitHub Actions concurrency feature #8546
    • Updated Makefile do_release script to include godoc steps #8550
    • Changing codahale/hdrhistogram import path #8637
    • Upgrade to Go 1.17 #8640
    • Fixes for reparent endtoend test flakiness #8642
    • hooks: remove govet because it already runs as part of golangci-lint #8674
    • Enhancement of the release instructions #8790
    • Remove consul-api usage in favor of official consul/api #8794
    • Enhancement of the release notes generation #8877

    Governance

    • update MAINTAINERS and CODEOWNERS for deepthi, pH14 and vmg #8675

    Query Serving

    • Cleanup: import of gh-ost test suite is complete #8459

    VReplication

    • VReplication support for non-PRIMARY KEY (Online DDL context) #8364

    Documentation

    Build/CI

    • maintainers: add Messaging as area of expertise #8578

    Cluster management

    • Enhance k8stopo flag documentation #8464
    • Update some vtctl/vtctlclient command help #8560
    • Trivial: fix the GenerateShardRanges vtctl[client] help #8586

    Examples

    • examples: update README to use new vreplication command syntax #8825

    Governance

    • update governance #8609

    Query Serving

    • vtexplain examples #8652

    VReplication

    • vtctl help/usage update #8879

    VTAdmin

    • [vtadmin] Remove outdated commands from vtadmin-web README #8496
    • Initial pass at some vtadmin docs #8526

    Enhancement

    Backup and Restore

    • Add Support for Restoring Specific Backups #8824

    Build/CI

    • Initial support for stress testing in end-to-end tests #8406
    • CI: Remove mariadb101 which is well past eol and no longer available in distributions #8446
    • Adding GitHub Self Hosted Runner tests #8721

    Cluster management

    • Vitess mixin improvements #7499
    • Expose topo_consul_lock_session_checks to allow customized consul session check #8298
    • Handle lock release with SIGHUP in VTGR #8472
    • Enhance PRS error message #8529
    • srvtopo: resilient watcher #8583
    • Add -restart_before_backup parameter for vtbackup #8608
    • srvtopo: allow unwatching from watch callbacks #8633
    • introduced cluster_operation as a new error code in vtrpc #8646
    • servenv: add --onclose_timeout flag #8651
    • Improve determinism of bootstrap #8840
    • OnlineDDL in v12: remove legacy code deprecated in earlier version #8972

    Observability

    • [vtadmin] cluster debug pages #8427
    • Export memstats for statsd backend #8777

    Query Serving

    • [tablet, queryrules] Extend query rules to check comments #8233
    • Add "show global status" support #8344
    • added batch lookup param to lookup vindex #8398
    • Online DDL/VReplication: able to read from replica #8405
    • multishard autocommit should work for bypass routing #8428
    • Add "no-scatter" flag to prohibit the use of scatter queries #8439
    • OnlineDDL: -skip-topo is always 'true' #8450
    • Add rows as keyword in sqlparser #8467
    • Periodic update of gh-ost binary #8470
    • fix when show tables record too much error log "Got unhandled packet.." #8478
    • SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8493
    • NativeDDL: analyzing added&removed unique keys #8495
    • Refactor authentication server plugin mechanism #8503
    • Update to planetscale/tengo v0.10.1-ps.v4 #8516
    • query serving to continue when topo server restarts - main #8534
    • support grpc reflection on APIs #8551
    • Customized CreateLookUpVindex to be Compatible with non-consistent Lookup Vindex #8570
    • Add additional options for configuring SSL modes as a client #8588
    • Fail plan for unsupported aggregate function #8593
    • gateway: use keyspace events when buffering requests #8700
    • Add optional query annotations, i.e. prefix SQL comments indicating the #8783
    • Slight improvement of the ACL error message #8805
    • Changed parsing for unions #8821
    • Use vt_dba user for online schema migration cutover phase #8836
    • Add parsing support for column list with subquery alias #8884
    • Add planner-version flag to vtexplain #8979
    • [Backport 12.0]: Add planner-version flag to vtexplain #9003

    TabletManager

    • Allow min TLS version for tablet to mysqld conns #8757

    VReplication

    • VReplication: Add ability to tag workflows #8388
    • Ignore SBR statements from pt-table-checksum #8396
    • Tablet Picker: add metric to record lack of available tablets #8403
    • VReplication: support different keys on source and target tables (different covered columns) #8423
    • VStream API: handle reparenting and unhealthy tablets #8445
    • Change local example to use v2 vreplication flows and change default to v2 from v1 #8553
    • Customized CreateLookUpVindex to Include a Flag to Continue Vreplication After Backfill with an Owner Provided #8572
    • Update MoveTables CLI help to reflect change we made at the beginning of #8597
    • VStreamer Field/Row Events: add Keyspace/Shard #8598
    • VStream API: Add flag to stop streaming on a reshard #8628

    VTAdmin

    • [vtadmin] cluster rpc pools #8421
    • Add support for passing custom interceptors to vtadmin grpcserver #8507

    Feature Request

    Cluster management

    • [vtctldserver] Migrate remaining ServingGraph rpcs to VtctldServer #8249
    • [grpctmclient] Add support for (bounded) per-host connection reuse #8368
    • VTGR: Vitess + MySQL group replication #8387
    • Add isActive flag to vtgr to support multi-cell topology #8780
    • [vtctld] Add SetWritable, StartReplication and StopReplication rpcs #8816
    • [vtctld] sleep/ping tablets #8826
    • [vtctld] migrate more util rpcs #8843
    • [vtctld] migrate validator rpcs #8849
    • [vtctld] localvtctldclient #8882
    • [vtctldserver] Migrate RunHealthCheck #8892
    • [vtctl] run new commands as standalone binary #8893

    VTAdmin

    • [vtadmin] [experimental] add per-api RBAC implementation #8515
    • [vtadmin] shard replication positions #8775
    • [vtadmin] GetVtctlds #8792

    Internal Cleanup

    Build/CI

    • Bump aws-sdk-go to v1.34.2 #8632
    • Upgrade consul api: go get github.com/hashicorp/consul/[email protected] #8784

    Cluster management

    • topodata: remove deprecated field 'ServedTypes' #8566
    • Upgrade to k8s 1.18 client #8762
    • [vtctl] command deprecations #8967

    Deployments

    • Remove the deprecated Helm charts and related code #8868

    Examples

    • Bump ini from 1.3.5 to 1.3.8 in /vitess-mixin/e2e #8823
    • backport operator updates to v12 branch . #9057

    General

    • Migrate k8s topo CRD to v1 api (v12 branch) #9046

    Governance

    • fix copyright #8611

    Query Serving

    • Adds flag to vttablet to disallow online DDL statements #8433
    • Remove vtrpc.LegacyErrorCode #8456
    • Allow for configuration of the minimal TLS version #8460
    • engine: allow retrying partial primitives #8727
    • srvtopo: expose WatchSrvKeyspace #8752
    • Clean up Primitive interface implementations #8901

    VReplication

    • LegacySplitCloneWorker is no longer used in any code paths #8867

    VTAdmin

    • [vtadmin] Add Options struct to wrap grpc/http options #8461
    • Update vtadmin local scripts to enable basic rbac #8801

    vtctl

    • Correctly identify backup timestamp var as a string #8891

    Other

    Examples

    • Improve the Docker local and compose examples #8685

    Performance

    Cluster management

    • throttler: don't allocate any resources unless it is actually enabled #8643

    Query Serving

    • cache: track memory usage more closely #8804

    VTAdmin

    • Use fmt.Fprintf instead of fmt.Sprintf #8922

    Testing

    Cluster management

    • Alter tests to reuse cluster components #8276
    • Adds some more orchestrator tests to vtorc #8535

    Query Serving

    • increase conn killer check to double the tx timeout value #8649
    • Added UNION testcases and auxilary code for running tests #8797
    • fixed regression in v3 for grouping by integer functions #8856

    VTAdmin

    • [vtadmin] Add a vtctld Dialer unit test #8455

    The release includes 1351 commits (excluding merges)

    Thanks to all our contributors: @Anders-PlanetScale, @GuptaManan100, @Johnny-Three, @Juneezee, @ajm188, @aquarapid, @askdba, @bnu0, @carsonoid, @choo-stripe, @dbussink, @dctrwatson, @deepthi, @demmer, @dependabot[bot], @derekperkins, @doeg, @eeSeeGee, @falun, @fatih, @frouioui, @hallaroo, @harshit-gangal, @ilikeorangutans, @mattlord, @rafael, @ritwizsinha, @rohit-nayak-ps, @sahutd, @shlomi-noach, @sonne5, @systay, @tdakkota, @tokikanno, @utk9, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.0-de06fca.tar.gz(400.35 MB)
    vitess-12.0.0-de06fca.x86_64.rpm(398.64 MB)
    vitess_12.0.0-de06fca_amd64.deb(400.36 MB)
  • v12.0.0-rc1(Oct 5, 2021)

    Release of Vitess v12.0.0

    Major Changes

    This release includes the following major changes or new features.

    Inclusive Naming

    A number of commands and RPCs have been deprecated as we move from master to primary. All functionality is backward compatible except as noted under Incompatible Changes.

    Gen4 Planner

    The newest version of the query planner, Gen4, becomes an experimental feature as part of this release. While Gen4 has been under development for a few release cycles, we have now reached parity with its predecessor, v3.

    To use Gen4, VTGate's -planner_version flag needs to be set to gen4.

    Minor Changes

    Query Serving

    • Add SHOW VITESS_REPLICATION_STATUS Query Support #8900
      • The new SHOW VITESS_REPLICATION_STATUS command/query shows the MySQL replica/replication (not vreplication) health for the vitess deployment.

    Incompatible Changes

    vtctl command output

    Wherever vtctl commands produced master or MASTER for tablet type, they now produce primary or PRIMARY. Scripts and tools that depend on parsing command output will need to change.

    Example:

    $ vtctlclient -server localhost:15999 ListAllTablets 
    zone1-0000000100 sourcekeyspace 0 primary 192.168.0.134:15100 192.168.0.134:17100 [] 2021-09-24T01:12:00Z
    zone1-0000000101 sourcekeyspace 0 rdonly 192.168.0.134:15101 192.168.0.134:17101 [] <null>
    zone1-0000000102 sourcekeyspace 0 replica 192.168.0.134:15102 192.168.0.134:17102 [] <null>
    zone1-0000000103 sourcekeyspace 0 rdonly 192.168.0.134:15103 192.168.0.134:17103 [] <null>
    

    Governance

    • Added @GuptaManan100 to maintainers #8833

    Changelog

    Bug fixes

    Build/CI

    • Update golang/x dependencies #8688
    • tests: use AtomicInt32 instead of int to fix races #8696

    Cluster management

    • Use consul lock properly #8310
    • Trivial: mysqlctl reinit_config -h was not showing help #8369
    • Port #8422 to main branch #8745
    • [vtctl] Add missing call to flag.Parse in commandGetRoutingRules #8795
    • Fix for padding in OrderAndCheckPartitions #8873

    Observability

    • schema engine: reset table size stats when a table is dropped #8634

    Query Serving

    • Fix for function calls in DEFAULT value of CREATE TABLE statement in main #8477
    • Fixing multiple issues related to onlineddl/lifecycle #8500
    • boolean values should not be parenthesised in default clause #8502
    • mysql/server: abort connection if secure transport is required #8504
    • Clear only the current shard's schemacopy rows during schema reload #8519
    • go/mysql/auth: fix error message for wrong auth method. #8525
    • Fix handshake protocol with MySQL client #8537
    • Copy parameter data from byte buffer in COM_STMT_SEND_LONG_DATA #8562
    • Negative float default #8587
    • [vtexplain] prevent panic on show tables/show full tables #8601
    • OnlineDDL: better scheduling/cancellation logic #8603
    • Fix parsing of FIRST clause in ALTER TABLE #8614
    • onlineddl Executor: build schema with DBA user #8624
    • Handle subquery merging with references correctly #8662
    • Fixing error on deletes from owning table for a lookup being populated by CreateLookupVindex after a SwitchWrites #8701
    • Fixing a panic in vtgate with OLAP mode #8722
    • default to primary tablet if not set in VStream api #8755
    • Fix bug in parsing of version comments #8770
    • Proper merge of the SysTableTableName fields when joining two routes #8771
    • fix parsing of MySQL Server Version flag #8791
    • Fix typo in pool config syntax. Fixes #8819 #8820
    • VtGate schema tracker to use provided user #8857
    • fix merging dba query as a subquery both in v3 and gen4 #8871

    VReplication

    • MoveTables: don't create unnecessary streams on the target for non-intersecting sources and targets #8090
    • VDiff: Add BIT datatype to list of byte comparable types #8401
    • Fix vreplication error metric #8483
    • Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
    • Fix VReplication logging to file and db #8521
    • Fix passing the wrong cell/cells variable to CreateLookupVindex #8590
    • Refresh SrvVSchema after an ExternalizeVindex: was missing #8670
    • Fixing missing argument in function call #8692
    • Fix how we identify MySQL generated columns #8763

    VTAdmin

    • [vtadmin-web] Set Backup status indicators to the correct colours #8410
    • [vtadmin] Fix bad copy-paste in pool config flag parsing #8518

    vtctl

    • Print actual current datetime for vtctl dry run commands #8778

    vttestserver

    • Remove lingering mysqld sock files on vttestserver container startup #8463
    • Fixed filename for the configuration file in Vttestserver #8809

    CI/Build

    Build/CI

    • Update release process #7759
    • [ci] Add errcheck to golangci-lint #7996
    • gomod: do not replace GRPC #8416
    • consolidation: Fix flaky test #8417
    • Small build improvements #8418
    • proto: upgrade vtprotobuf version #8454
    • Automatically use the latest tag of Vitess for cluster upgrade E2E test #8471
    • Addition of known issues to release notes #8482
    • Fix Cluster 14 flakiness #8494
    • add question on backporting back to PR template #8541
    • Enable GitHub Actions concurrency feature #8546
    • Updated Makefile do_release script to include godoc steps #8550
    • Changing codahale/hdrhistogram import path #8637
    • Upgrade to Go 1.17 #8640
    • Fixes for reparent endtoend test flakiness #8642
    • hooks: remove govet because it already runs as part of golangci-lint #8674
    • Enhancement of the release instructions #8790
    • Remove consul-api usage in favor of official consul/api #8794
    • Enhancement of the release notes generation #8877

    Governance

    • update MAINTAINERS and CODEOWNERS for deepthi, pH14 and vmg #8675

    Query Serving

    • Cleanup: import of gh-ost test suite is complete #8459

    VReplication

    • VReplication support for non-PRIMARY KEY (Online DDL context) #8364

    Documentation

    Build/CI

    • maintainers: add Messaging as area of expertise #8578

    Cluster management

    • Enhance k8stopo flag documentation #8464
    • Update some vtctl/vtctlclient command help #8560
    • Trivial: fix the GenerateShardRanges vtctl[client] help #8586

    Examples

    • examples: update README to use new vreplication command syntax #8825

    Governance

    • update governance #8609

    Query Serving

    • vtexplain examples #8652

    VReplication

    • vtctl help/usage update #8879

    VTAdmin

    • [vtadmin] Remove outdated commands from vtadmin-web README #8496
    • Initial pass at some vtadmin docs #8526

    Enhancement

    Backup and Restore

    • Add Support for Restoring Specific Backups #8824

    Build/CI

    • Initial support for stress testing in end-to-end tests #8406
    • CI: Remove mariadb101 which is well past eol and no longer available in distributions #8446
    • Adding GitHub Self Hosted Runner tests #8721

    Cluster management

    • Vitess mixin improvements #7499
    • Expose topo_consul_lock_session_checks to allow customized consul session check #8298
    • Handle lock release with SIGHUP in VTGR #8472
    • Enhance PRS error message #8529
    • srvtopo: resilient watcher #8583
    • Add -restart_before_backup parameter for vtbackup #8608
    • srvtopo: allow unwatching from watch callbacks #8633
    • introduced cluster_operation as a new error code in vtrpc #8646
    • servenv: add --onclose_timeout flag #8651
    • Improve determinism of bootstrap #8840

    Observability

    • [vtadmin] cluster debug pages #8427
    • Export memstats for statsd backend #8777

    Query Serving

    • [tablet, queryrules] Extend query rules to check comments #8233
    • Add "show global status" support #8344
    • added batch lookup param to lookup vindex #8398
    • Online DDL/VReplication: able to read from replica #8405
    • multishard autocommit should work for bypass routing #8428
    • Add "no-scatter" flag to prohibit the use of scatter queries #8439
    • OnlineDDL: -skip-topo is always 'true' #8450
    • Add rows as keyword in sqlparser #8467
    • Periodic update of gh-ost binary #8470
    • fix when show tables record too much error log "Got unhandled packet.." #8478
    • SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8493
    • NativeDDL: analyzing added&removed unique keys #8495
    • Refactor authentication server plugin mechanism #8503
    • Update to planetscale/tengo v0.10.1-ps.v4 #8516
    • query serving to continue when topo server restarts - main #8534
    • support grpc reflection on APIs #8551
    • Customized CreateLookUpVindex to be Compatible with non-consistent Lookup Vindex #8570
    • Add additional options for configuring SSL modes as a client #8588
    • Fail plan for unsupported aggregate function #8593
    • gateway: use keyspace events when buffering requests #8700
    • Add optional query annotations, i.e. prefix SQL comments indicating the #8783
    • Slight improvement of the ACL error message #8805
    • Changed parsing for unions #8821
    • Use vt_dba user for online schema migration cutover phase #8836
    • Add parsing support for column list with subquery alias #8884

    TabletManager

    • Allow min TLS version for tablet to mysqld conns #8757

    VReplication

    • VReplication: Add ability to tag workflows #8388
    • Ignore SBR statements from pt-table-checksum #8396
    • Tablet Picker: add metric to record lack of available tablets #8403
    • VReplication: support different keys on source and target tables (different covered columns) #8423
    • VStream API: handle reparenting and unhealthy tablets #8445
    • Change local example to use v2 vreplication flows and change default to v2 from v1 #8553
    • Customized CreateLookUpVindex to Include a Flag to Continue Vreplication After Backfill with an Owner Provided #8572
    • Update MoveTables CLI help to reflect change we made at the beginning of #8597
    • VStreamer Field/Row Events: add Keyspace/Shard #8598
    • VStream API: Add flag to stop streaming on a reshard #8628

    VTAdmin

    • [vtadmin] cluster rpc pools #8421
    • Add support for passing custom interceptors to vtadmin grpcserver #8507

    Feature Request

    Cluster management

    • [vtctldserver] Migrate remaining ServingGraph rpcs to VtctldServer #8249
    • [grpctmclient] Add support for (bounded) per-host connection reuse #8368
    • VTGR: Vitess + MySQL group replication #8387
    • Add isActive flag to vtgr to support multi-cell topology #8780
    • [vtctld] Add SetWritable, StartReplication and StopReplication rpcs #8816
    • [vtctld] sleep/ping tablets #8826
    • [vtctld] migrate more util rpcs #8843
    • [vtctld] migrate validator rpcs #8849
    • [vtctld] localvtctldclient #8882
    • [vtctldserver] Migrate RunHealthCheck #8892
    • [vtctl] run new commands as standalone binary #8893

    VTAdmin

    • [vtadmin] [experimental] add per-api RBAC implementation #8515
    • [vtadmin] shard replication positions #8775
    • [vtadmin] GetVtctlds #8792

    Internal Cleanup

    Build/CI

    • Bump aws-sdk-go to v1.34.2 #8632
    • Upgrade consul api: go get github.com/hashicorp/consul/[email protected] #8784

    Cluster management

    • topodata: remove deprecated field 'ServedTypes' #8566
    • Upgrade to k8s 1.18 client #8762

    Deployments

    • Remove the deprecated Helm charts and related code #8868

    Examples

    • Bump ini from 1.3.5 to 1.3.8 in /vitess-mixin/e2e #8823

    Governance

    • fix copyright #8611

    Query Serving

    • Adds flag to vttablet to disallow online DDL statements #8433
    • Remove vtrpc.LegacyErrorCode #8456
    • Allow for configuration of the minimal TLS version #8460
    • engine: allow retrying partial primitives #8727
    • srvtopo: expose WatchSrvKeyspace #8752
    • Clean up Primitive interface implementations #8901

    VReplication

    • LegacySplitCloneWorker is no longer used in any code paths #8867

    VTAdmin

    • [vtadmin] Add Options struct to wrap grpc/http options #8461
    • Update vtadmin local scripts to enable basic rbac #8801

    vtctl

    • Correctly identify backup timestamp var as a string #8891

    Other

    Examples

    • Improve the Docker local and compose examples #8685

    Performance

    Cluster management

    • throttler: don't allocate any resources unless it is actually enabled #8643

    Query Serving

    • cache: track memory usage more closely #8804

    VTAdmin

    • Use fmt.Fprintf instead of fmt.Sprintf #8922

    Testing

    Cluster management

    • Alter tests to reuse cluster components #8276
    • Adds some more orchestrator tests to vtorc #8535

    Query Serving

    • increase conn killer check to double the tx timeout value #8649
    • Added UNION testcases and auxilary code for running tests #8797
    • fixed regression in v3 for grouping by integer functions #8856

    VTAdmin

    • [vtadmin] Add a vtctld Dialer unit test #8455

    The release includes 1281 commits (excluding merges)

    Thanks to all our contributors: @Anders-PlanetScale, @GuptaManan100, @Johnny-Three, @Juneezee, @ajm188, @aquarapid, @askdba, @bnu0, @choo-stripe, @dbussink, @dctrwatson, @deepthi, @demmer, @dependabot[bot], @derekperkins, @doeg, @eeSeeGee, @falun, @fatih, @frouioui, @hallaroo, @harshit-gangal, @ilikeorangutans, @mattlord, @rafael, @ritwizsinha, @rohit-nayak-ps, @sahutd, @shlomi-noach, @sonne5, @systay, @tdakkota, @tokikanno, @utk9, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.0-rc1-bd0a4ed.tar.gz(401.79 MB)
    vitess-12.0.0_rc1-bd0a4ed.x86_64.rpm(400.06 MB)
    vitess_12.0.0-rc1-bd0a4ed_amd64.deb(401.79 MB)
  • v11.0.1(Sep 9, 2021)

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v11.0.1, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v11.0.4 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Bug fixes

    Cluster management

    • Fix problematic ERS cases #8744

    Query Serving

    • Handle subquery merging with references correctly #8661
    • onlineddl Executor: build schema with DBA user #8667
    • Backport to 11: Fixing a panic in vtgate with OLAP mode #8746
    • Backport into 11: default to primary tablet if not set in VStream api #8766

    VReplication

    • Refresh SrvVSchema after an ExternalizeVindex: was missing #8669

    CI/Build

    Build/CI

    • Vitess Release 11.0.0 #8549
    • Backport to 11: Updated Makefile do_release script to include godoc steps #8787

    The release includes 18 commits (excluding merges) Thanks to all our contributors: @aquarapid, @askdba, @frouioui, @harshit-gangal, @rohit-nayak-ps, @shlomi-noach, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.1-92ac1ff.tar.gz(397.08 MB)
    vitess-11.0.1-92ac1ff.x86_64.rpm(394.80 MB)
    vitess_11.0.1-92ac1ff_amd64.deb(397.09 MB)
  • v11.0.0(Jul 27, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v11.0.0, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v11.0.4 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Bug fixes

    Build/CI

    • update moby/term to fix darwin build issue #7787
    • Removing Linux/amd64 specific subdependencies #7796
    • CI: run some tests inside docker to workaround GH Actions issue #7868
    • Fix flaky race condition in vtexplain #7930
    • Fixing flaky endtoend/tablegc tests #7947
    • Add libwww-perl to allow pt-osc wrapper to work in vitess/lite #8141

    Cluster management

    • Add missing return on a failed GetShard call in FindAllShardsInKeyspace #7992
    • Increase the default srv_topo_timeout #8011
    • parser: support index name in FOREIGN KEY clause; Online DDL to reject FK clauses #8058
    • Correctly parse cell names with dashes in tablet aliases #8167
    • /healthz should not report ok when vttablet is not connected to mysql #8238
    • [topo] Refactor ExpandCells to not error on valid aliases #8291
    • tm state: don't populate metadata in updateLocked #8362
    • [grpcvtctldserver] Fix backup detail limit math #8402

    Query Serving

    • fix select star, col1 orderby col1 bug. #7743
    • VTExplain: Add support for multi table join query #7829
    • Routing of Information Schema Queries #7841
    • Fix + integration test for keyspaces_to_watch routing regression [fixes #7882] #7873
    • Revert "[tablet, queryrules] Extend query rules to check comments" #7897
    • vtctl: return error on invalid ddl_strategy #7923
    • Panic when EOF after @ symbol #7925
    • Fixes encoding of sql strings #8033
    • Make TwoPC engine initialization async. #8048
    • Fix for issue with information_schema queries with both table name and schema name predicates #8087
    • Fix for transactions not allowed to finish during PlannedReparentShard #8089
    • Set fully parsed to false when ignoring errors #8094
    • Fix for query and sub query with limits #8097
    • Fix buffering when using reserved connections #8102
    • Parse generated columns in DDL #8117
    • More explicit message on online DDL parsing error #8118
    • healthcheck: attempt to update primary only if the current tablet is serving #8121
    • PRIMARY in index hint list for master #8160
    • Signed int parse #8189
    • Delete table reference alias support #8393
    • Fix for function calls in DEFAULT value of CREATE TABLE statement in release-11.0 #8476
    • Backport: Fixing multiple issues related to onlineddl/lifecycle #8517
    • boolean values should not be parenthesised in default clause - release 11 #8531

    VReplication

    • VDiff: Use byte compare if weight_string() returns null for either source or target #7696
    • Rowlog: Update rowlog for the API change made for the vstream skew alignment feature #7809
    • Pad binlog values for binary() columns to match the value returned by mysql selects #7969
    • Change vreplication error metric name to start with a string to pass prometheus validations #7983
    • Pass the provided keyspace through to UpdateDisableQueryService rather than hard-coding the sourceKeyspace #8020
    • Fix vreplication timing metrics #8024
    • Switchwrites: error if no tablets available on target for reverse replication #8142
    • Remove noisy vexec logs #8144
    • VReplicationExec: don't create new stats objects on a select #8166
    • Binlog JSON Parser: handle inline types correctly for large documents #8187
    • Schema Tracking Flaky Test: Ignore unrelated gtid progress events #8283
    • Adds padding to keyrange comparison #8296
    • VReplication Reverse Workflows: add keyspace scope to vindex while creating reverse vreplication streams #8385
    • OnlineDDL/Vreplication stress test: investigating failures #8390
    • Ignore SBR statements from pt-table-checksum #8396
    • Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
    • VDiff: Add BIT datatype to list of byte comparable types #8401
    • Fix excessive VReplication logging to file and db #8521

    VTAdmin

    • Add missing return in vtctld-* DSN case, and log any flag that gets ignored #7872
    • [vtadmin-web] Do not parse numbers/booleans in URL query parameters by default #8100
    • [vtadmin-web] Small bugfix where stream source displayed instead of target #8311

    CI/Build

    Build/CI

    • Planbuilder: Fix fuzzer #7952
    • java: Bump SNAPSHOT version to 11.0.0-SNAPSHOT after Vitess release v10 #7968
    • Add optional TLS feature to gRPC servers #8049
    • Add image and base image arguments to build.sh #8064
    • [fuzzing] Add report #8128
    • CI: fail PR if it does not have required labels #8147
    • trigger pr-labels workflow when labels added/removed from PR #8157
    • Add vmg as a maintainer #8186
    • Refactor vtorc endtoend tests #8215
    • Use shared testing helper which guards against races in tmclient construction #8300
    • moved multiple vtgate and tabletgateway to individual shards #8305
    • include squashed PRs in release notes #8326
    • pr-labels workflow: only check actual PRs #8327
    • Heuristic fix for a flaky test: allow for some noise to pass through #8339
    • ci: upgrade to Go 1.16 #8274
    • mod: upgrade etcd to stable version #8357
    • Resolve go.mod issue with direct dependency on planetscale/tengo #8383
    • Initial GitHub Docker Build Setup #8399

    Cluster management

    • make: build vitess as static binaries by default #7795
    • Fix TableGC flaky test by reducing check interval #8270

    Java

    • Bump commons-io from 2.6 to 2.7 in /java #7953

    Other

    • Bump MySQL version for docker builds to 8.0.23 #7811

    Query Serving

    • Online DDL Vreplication test suite: adding tests #8213
    • Online DDL/VReplication: more passing tests (no UK/PK) #8225
    • Online DDL: reject ALTER TABLE...RENAME statements #8227
    • Online DDL/VReplication: fail on existence of FOREIGN KEY #8228
    • Online DDL/VReplication: test PK column case change #8336
    • Online DDL/VReplication: add PK DATETIME->TIMESTAMP test #8338
    • Bugfix: assign cexpr.references for column CONVERTed to utf8mb4 #8355

    VReplication

    • Towards a VReplication/OnlineDDL testing suite #8181
    • Online DDL/Vreplication: column type awareness #8239
    • remove duplicate ReadMigrations, fixing build #8258
    • VReplication (and by product, Online DDL): support GENERATED column as part of PRIMARY KEY #8335

    vttestserver

    • docker/vttestserver: Set max_connections in default-fast.cnf at container build time #7810

    Documentation

    Cluster management

    • Enhance k8stopo flag documentation #8458

    Build/CI

    • Update version for latest snapshot #7801
    • v10 GA Release Notes #7964
    • Updates and Corrections to v9 Release Notes #8295

    Query Serving

    • Update link to 'Why FK not supported in Online DDL' blog post #8372

    Other

    • correct several wrong words #7822
    • MAINTAINERS.md: Update enisoc's email. #7909
    • Modification of the Markdown list format in the release notes template #7962
    • Update vreplicator docs #8014
    • Release notes: add Known Issues #8027

    Enhancement

    Observability

    • [trace] Add logging support to opentracing plugins #8289

    Build/CI

    • Makefile: add cross-build target for cross-compiling client binaries #7806
    • git: improve signoff detection #7852
    • Release notes generation #7932
    • vitess/lite docker build fails for mysql80 #7943
    • Fix a gofmt warning #7959
    • docker/vttestserver/run.sh: Add $CHARSET environment variable #7970
    • docker/lite/install_dependencies.sh: If the dependency install loop reaches its max number of retries, consider that a failure; exit the script nonzero so the build halts. #7976
    • Add commit count and authors to the release notes #7982
    • Automate version naming and release tagging #8034
    • Make sure to only allow codegen when the rest of the code compiles #8169
    • Add options to vttestserver to pass -foreign_key_mode, -enable_online_ddl, and -enable_direct_ddl through to vtcombo #8177

    Cluster management

    • Dynamic throttle metric threshold #7742
    • Bump Bootstrap version, per CVE-2018-14040 (orchestrator/vtorc) #7824
    • Check tablet alias before removing after error stream #7915
    • vttablet/tabletmanager - add additional test for tmstate.Open() #7993
    • Add vttablet_restore_done hook #8007
    • Added ValidateVSchema #8012
    • Add RemoteOperationTimeout to both legacy and grpc ChangeTabletType implementations. #8052
    • [vtctldserver] Add guard against self-reparent, plus misc updates #8084
    • [tm_state] updateLocked should re-populate local metadata tables to reflect promotion rule changes #8107
    • [vtctld] Migrate ApplyVSchema to VtctldServer #8113
    • vtctl.generateShardRanges -> key.GenerateShardRanges #8134
    • etcd: add grpc.WithBlock to client config #8205
    • [workflow] Add tracing to GetWorkflows endpoint #8266
    • [vtctldclient] Add legacy shim #8284
    • [vtctldserver] Add tracing #8285
    • [vtctldserver] Add additional backup info fields #8321
    • [workflow] Call scanWorkflow concurrently #8272

    Query Serving

    • [Gen4] Implemented table alias functionality in the semantic analysis #7629
    • Improved error messages (tabletserver) #7747
    • Allow modification of tablet unhealthy_threshold via debugEnv #7753
    • [Gen4] Initial Horizon Planning #7775
    • [tablet, queryrules] Extend query rules to check comments #7784
    • Add support for showing global gtid executed per shard #7856
    • Minor cleanups around errors on the vtgate #7864
    • log unsupported queries #7865
    • Show databases like #7912
    • Add rank as reserved keyword #7944
    • Online DDL: introducing ddl_strategy -singleton-context flag #7946
    • Ignore the error and log as warn if not able to validate the current system setting for check and ignore case #8004
    • Detect and signal schema changes on vttablets #8005
    • DDL bypass plan #8013
    • Online DDL: progress & ETA for Vreplication migrations #8015
    • Scatter errors as warning in olap query #8018
    • livequeryz: livequeryz/terminate link should be relative #8025
    • Handle online DDL user creation if we do not have SUPER privs (e.g. AWS Aurora) #8038
    • Update the schema copy with minimal changes #8067
    • Gen4: Support for order by column not present in projection #8070
    • Schema tracking in vtgate #8074
    • protobuf: upgrade #8075
    • added mysql 8.0 reserved keywords #8086
    • Fix ghost/pt-osc in the external DB case where MySQL might be reporting #8115
    • Support for vtgate -enable_direct_ddl flag #8116
    • add transaction ID to query log for acquisition and analysis #8133
    • Inline reference #8136
    • Improve ScatterErrorsAsWarnings functionality #8139
    • Gen4: planning Select IN #8155
    • Primary key name #8188
    • Online DDL: read and publish gh-ost FATAL message where possible #8192
    • Expose inner net.Conn to be able to write better unit tests #8217
    • vtgate: validate important flags during startup #8218
    • Online DDL/VReplication: AUTO_INCREMENT support and tests #8223
    • [Gen4] some renaming from v4 to Gen4 #8234
    • Schema Tracking: Optimize and Bug Fix #8243
    • gen4: minor refactoring to improve readability #8245
    • gen4: plan more opcodes #8254
    • Online DDL: report rows_copied for migration (initial support in gh-ost) #8255
    • Schema tracking: new tables in sharded keyspace #8256
    • [parser] use table_alias for ENGINE option in CREATE TABLE stmt #8307
    • Add /debug/env for vtgate #8292
    • gen4: outer joins #8312
    • Gen4: expand star in projection list #8325
    • gen4: Fail all queries not handled well by gen4 #8359
    • Gen4 fail more2 #8382
    • SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8532
    • [11.0] query serving to continue when topo server restarts #8533
    • [11.0] Disable allowing set statements on system settings by default #8540

    VReplication

    • Change local example to use v2 vreplication flows and make v2 flows as the default #8527
    • Use Dba user when Vexec is runAsAdmin #7731
    • Add table for logging stream errors, state changes and key workflow steps #7831
    • Fix some static check warning #7960
    • VSchema Validation on ReshardWorkflow Creation #7977
    • Tracking rows_copied #7980
    • Vdiff formatting improvements #8079
    • Ignore generated columns in workflows #8129
    • VReplication Copy Phase: Increase default replica lag tolerance. Also make it and copy timeout modifiable via flags #8130
    • Materialize: Add additional comparison operators in Materialize and fix bug where they not applied for sharded keyspaces #8247
    • Copy Phase: turn on OptimizeInserts by default #8248
    • Tracker/VStreamer: only reload schema for tables in current database and not for internal table artifacts #8257
    • Online DDL/Vreplication suite: support ENUM->VARCHAR/TEXT type change #8275
    • Added TableName to DiffReport struct. #8279
    • New VReplication lag metric #8306
    • VStream API: add heartbeat for idle streams #8244
    • Online DDL/VReplication: support non-UTF8 character sets #8322
    • Online DDL/Vreplication suite: fix test for no shared UK #8334
    • Online DDL/VReplication: support DROP+ADD column of same name #8337
    • Online DDL/VReplication test suite: support ENUM as part of PRIMARY KEY #8345
    • Change local example to use v2 vreplication flows #8527
    • Tablet Picker: add metric to record lack of available tablets #8403

    VTAdmin

    • [vtadmin-web] Add useSyncedURLParam hook to persist filter parameter in the URL #7857
    • [vtadmin-web] Display more data on /gates view and add filtering #7876
    • [vtadmin] Promote ErrNoSchema to a TypedError which returns http 404 #7885
    • [vtadmin] gate/tablet/vtctld FQDNs #7886
    • [vtadmin-web] Display vindex data on Schema view #7917
    • [vtadmin-web] Add filtering and source/target shards to Workflows view #7948
    • [vtadmin-web] Add filtering + shard counts/status to Keyspaces view #7991
    • [vtadmin-web] Display shard state on Tablets view + extract tablet utilities #7999
    • [vtadmin] experimental tabletdebug #8003
    • [vtadmin-web] Display timestamps + stream counts on Workflows view #8009
    • [vtadmin-web] Updates to table styling + other CSS #8072
    • [vtadmin-web] Add Tab components #8119
    • [vtadmin-api] Update GetTablet to use alias instead of hostname #8163
    • [vtadmin-api] Rename flag 'http-tablet-fqdn-tmpl' to 'http-tablet-url-tmpl' + update vtadmin flags for local example #8164
    • Add --tracer flag to vtadmin and actually start tracing #8165
    • [vtadmin-web] The hastiest Tablet view (+ experimental debug vars) #8170
    • [vtadmin-web] Add tabs to Workflow view #8203
    • [vtctld] Add GetSrvVSchemas command #8221
    • [vtadmin-api] Add HTTP endpoints for /api/srvvschemas and /api/srvvschema/{cluster}/{cell} #8226
    • [vtadmin-web] Add QPS and VReplicationQPS charts to Tablet view #8263
    • [vtadmin-web] Add client-side error handling interface + Bugsnag implementation #8287
    • [vtadmin-web] Add chart for stream vreplication lag across all streams in a workflow #8331
    • [vtctldproxy] Add more annotations to vtctld Dial calls #8346

    vttestserver

    • Vttest create db #7989
    • Adds an environment variable to set the MySQL max connections limit in vttestserver docker image #8210

    Feature Request

    Build/CI

    • [trace] Add optional flags to support flexible jaeger configs #8199

    Cluster management

    • Add VtctldServer to vtcombo #7896
    • [vtctldserver] Migrate routing rules RPCs, and also RebuildVSchemaGraph #8197
    • [vtctldserver] Migrate CellInfo, CellAlias rw RPCs #8219
    • [vtctldserver] Add RefreshState RPCs #8232

    Query Serving

    • VTGate grpc implementation of Prepare and CloseSession #8211
    • Schema tracking: One schema load at a time per keyspace #8224

    VTAdmin

    • [vtadmin-web] Add DataFilter + Workspace layout components #8032
    • [vtadmin-web] Add Tooltip + HelpTooltip components #8076
    • [vtadmin-web] Add initial Stream view, render streams on Workflow view #8091
    • [vtadmin-web] The hastiest-ever VTExplain UI #8092
    • [vtadmin-web] Add source-map-explorer util #8093
    • [vtadmin-web] Add Keyspace detail view #8111
    • [vtadmin-api] Add GetKeyspace endpoint #8125
    • [workflow] Add vreplication_log data to workflow protos, and VtctldServer.GetWorkflows method #8261
    • [vtadmin] Add debug endpoints #8268

    Internal Cleanup

    Build/CI

    • Makefile: fix cross-build comments #8246
    • remove unused hooks with refs to master branch #8250
    • [vtctldserver] Update tests to prevent races during tmclient init #8320
    • sks-keyservers.net seems to be finally dead, replace with #8363
    • endtoend: change log level of etcd start from error to info #8370

    Cluster management

    • Online DDL: code cleanup #7589
    • [wrangler|topotools] Migrate UpdateShardRecords, RefreshTabletsByShard, and {Get,Save}RoutingRules #7965
    • Tear down old stream_migrater shim, now that we're fully in package workflow #8073
    • [mysqlctl] Restructure MetadataManager to reduce public API surface area #8152
    • naming: master to primary #8251
    • vtorc: code cleanup #8269

    Query Serving

    • Plan StreamExecute Queries #7941

    VReplication

    • [workflow] extract migration targets from wrangler #7934
    • [wrangler|workflow] Extract vrStream type to workflow.VReplicationStream #7966
    • [wrangler|workflow] Extract workflowState and workflowType out to package workflow #7967
    • [wrangler|workflow] extract *wrangler.streamMigrater to workflow.StreamMigrator #8008
    • [workflow] Migrate getCellsWith{Shard,Table}ReadsSwitched, TrafficSwitchDirection and TableRemovalType to package workflow #8190
    • [workflow] Cleanup wrangler wrappers, migrate checkIfJournalExistsOnTablet to package workflow #8193
    • Backports of #8403 #8483 #8489 #8401 #8521 #8396 from main into release 11.0 #8536

    VTAdmin

    • [vtadmin-api] Replace magic numbers with net/http constants #8127
    • [vtadmin-web] Move single-entity view components into subfolders #8202
    • [vtadmin] Ensure we log any errors when closing the tracer #8262

    Other

    Build/CI

    • add vtctldclient to binary directory #7889

    Cluster management

    • vttablet/tabletmanager: add isInSRVKeyspace/isShardServing #7929

    Other

    • Change VitessInputFormat key type #199
    • Online DDL plan via Send; "singleton" migrations on tablets #7785
    • flaky onlineddl tests: reduce -online_ddl_check_interval #7847
    • Looking into flaky endtoend upgrade test #7900
    • fixing flaky upgrade test #7901

    Query Serving

    • Introduce Concatenated Fixed-width Composite or CFC vindex #7537
    • Add common tags for stats backends that support it #7651
    • Add support for showing global vgtid executed #7797
    • perf: vttablet/mysql optimizations #7800
    • Fix bug with reserved connections to stale tablets #7879
    • Memory Sort to close the goroutines when callback returns error #7903
    • OnlineDDL: more migration check ticks upon migration start #7961
    • Update gh-ost binary to v1.1.3 #8021
    • VReplication Online DDL: fix classification of virtual columns #8043

    VReplication

    • VReplicationErrors metric: use . as delimiter instead of _ to behave well with Prometheus #7807

    VTAdmin

    • Update GetSchema filtering to exclude shards where IsMasterServing but no MasterAlias #7805
    • [vtadmin-api] Reintroduce include_non_serving_shards opt to GetSchema #7814
    • [vtadmin-web] Add DataCell component #7817
    • Rewrite useTableDefinitions hook as getTableDefinitions util #7821
    • [vtadmin-web] Display (approximate) table sizes + row count on /schemas view #7826
    • [vtadmin-web] Add Pip + TabletServingPip components #7827

    Performance

    Query Serving

    • vttablet: stream consolidation #7752
    • perf: optimize bind var generation #7828

    VReplication

    • Optimize the catchup phases by filtering out rows which not within range of copied pks during inserts #7708
    • Ability to compress gtid when stored in _vt.vreplication's pos column #7877
    • Performance & benchmarks (table copying) #7881
    • Dynamic packet sizing #7933
    • perf: vreplication client CPU usage #7951
    • VDIff: fix performance regression introduced by progress logging #8016
    • proto: Generate faster code using vtprotobuf #8173
    • proto: enable pooling for vreplication #8273

    Testing

    Build/CI

    • Attempt to fix TLS Server Flaky test #7842
    • TestMakeCommonTags Flaky Test: match elements #7843
    • upgrade tests: test against v9.0.0 #7848
    • FOSSA scan added #7862
    • ci: fix all racy tests #7904
    • CI: Revert docker change for unit tests from #7868 #7940
    • Add online ddl on start queries to schema list in vtexplain tablet #8397

    Java

    • [Java] JDBC mysql driver test #8154

    Other

    • Fuzzing: Fixup oss-fuzz build script #7782
    • Wrangler tests: Return a fake tablet in the wrangler test dialer to avoid tablet picker errors spamming the test logs #7863

    Query Serving

    • clean up test #7816
    • tabletserver: fix flaky test #7851
    • Planbuilder: Add fuzzer #7902
    • mysql: Small adjustments to fuzzer #7907
    • vtgate/engine: Add fuzzer #7914
    • Adding Fuzzer Test Cases #8106
    • Addition of fuzzer issues #8195

    VReplication

    • vstreamer: Add fuzzer #7918

    vttestserver

    • Speedup new vttestserver tests #8229
    • Vttestserver docker test #8253

    Cluster management

    • Make timestamp authoritative for master information #8381

    The release includes 1080 commits (excluding merges)

    Thanks to all our contributors: @AdamKorcz, @GuptaManan100, @Hellcatlk, @Johnny-Three, @acharisshopify, @ajm188, @alexrs, @aquarapid, @askdba, @deepthi, @doeg, @dyv, @enisoc, @frouioui, @gedgar, @guidoiaquinti, @harshit-gangal, @hkdsun, @idvoretskyi, @jmoldow, @kirs, @mcronce, @narcsfz, @noxiouz, @rafael, @rohit-nayak-ps, @setassociative, @shlomi-noach, @systay, @tokikanno, @vmg, @wangmeng99, @yangxuanjia, @zhangshj-inspur

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.0-aa798b8.tar.gz(397.07 MB)
    vitess-11.0.0-aa798b8.x86_64.rpm(394.82 MB)
    vitess_11.0.0-aa798b8_amd64.deb(397.08 MB)
  • v11.0.0-rc1(Jul 6, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Bug fixes

    Build/CI

    • update moby/term to fix darwin build issue #7787
    • Removing Linux/amd64 specific subdependencies #7796
    • CI: run some tests inside docker to workaround GH Actions issue #7868
    • Fix flaky race condition in vtexplain #7930
    • Fixing flaky endtoend/tablegc tests #7947
    • Add libwww-perl to allow pt-osc wrapper to work in vitess/lite #8141

    Cluster management

    • Add missing return on a failed GetShard call in FindAllShardsInKeyspace #7992
    • Increase the default srv_topo_timeout #8011
    • parser: support index name in FOREIGN KEY clause; Online DDL to reject FK clauses #8058
    • Correctly parse cell names with dashes in tablet aliases #8167
    • /healthz should not report ok when vttablet is not connected to mysql #8238
    • [topo] Refactor ExpandCells to not error on valid aliases #8291
    • tm state: don't populate metadata in updateLocked #8362
    • [grpcvtctldserver] Fix backup detail limit math #8402

    Query Serving

    • fix select star, col1 orderby col1 bug. #7743
    • VTExplain: Add support for multi table join query #7829
    • Routing of Information Schema Queries #7841
    • Fix + integration test for keyspaces_to_watch routing regression [fixes #7882] #7873
    • Revert "[tablet, queryrules] Extend query rules to check comments" #7897
    • vtctl: return error on invalid ddl_strategy #7923
    • Panic when EOF after @ symbol #7925
    • Fixes encoding of sql strings #8033
    • Make TwoPC engine initialization async. #8048
    • Fix for issue with information_schema queries with both table name and schema name predicates #8087
    • Fix for transactions not allowed to finish during PlannedReparentShard #8089
    • Set fully parsed to false when ignoring errors #8094
    • Fix for query and sub query with limits #8097
    • Fix buffering when using reserved connections #8102
    • Parse generated columns in DDL #8117
    • More explicit message on online DDL parsing error #8118
    • healthcheck: attempt to update primary only if the current tablet is serving #8121
    • PRIMARY in index hint list for master #8160
    • Signed int parse #8189
    • Delete table reference alias support #8393

    VReplication

    • VDiff: Use byte compare if weight_string() returns null for either source or target #7696
    • Rowlog: Update rowlog for the API change made for the vstream skew alignment feature #7809
    • VReplication: Pad binlog values for binary() columns to match the value returned by mysql selects #7969
    • Change vreplication error metric name to start with a string to pass prometheus validations #7983
    • Pass the provided keyspace through to UpdateDisableQueryService rather than hard-coding the sourceKeyspace #8020
    • vreplication: fix vreplication timing metrics #8024
    • Switchwrites: error if no tablets available on target for reverse replication #8142
    • VReplication: remove noisy vexec logs #8144
    • VReplicationExec: don't create new stats objects on a select #8166
    • Binlog JSON Parser: handle inline types correctly for large documents #8187
    • Schema Tracking Flaky Test: Ignore unrelated gtid progress events #8283
    • Adds padding to keyrange comparison #8296
    • VReplication Reverse Workflows: add keyspace scope to vindex while creating reverse vreplication streams #8385

    VTAdmin

    • Add missing return in vtctld-* DSN case, and log any flag that gets ignored #7872
    • [vtadmin-web] Do not parse numbers/booleans in URL query parameters by default #8100
    • [vtadmin-web] Small bugfix where stream source displayed instead of target #8311

    CI/Build

    Build/CI

    • Planbuilder: Fix fuzzer #7952
    • java: Bump SNAPSHOT version to 11.0.0-SNAPSHOT after Vitess release v10 #7968
    • Add optional TLS feature to gRPC servers #8049
    • Add image and base image arguments to build.sh #8064
    • [fuzzing] Add report #8128
    • CI: fail PR if it does not have required labels #8147
    • trigger pr-labels workflow when labels added/removed from PR #8157
    • Add vmg as a maintainer #8186
    • Refactor vtorc endtoend tests #8215
    • Use shared testing helper which guards against races in tmclient construction #8300
    • moved multiple vtgate and tabletgateway to individual shards #8305
    • include squashed PRs in release notes #8326
    • pr-labels workflow: only check actual PRs #8327
    • Heuristic fix for a flaky test: allow for some noise to pass through #8339
    • ci: upgrade to Go 1.16 #8274
    • mod: upgrade etcd to stable version #8357
    • Resolve go.mod issue with direct dependency on planetscale/tengo #8383
    • Initial GitHub Docker Build Setup #8399

    Cluster management

    • make: build vitess as static binaries by default #7795
    • Fix TableGC flaky test by reducing check interval #8270

    Java

    • Bump commons-io from 2.6 to 2.7 in /java #7953

    Other

    • Bump MySQL version for docker builds to 8.0.23 #7811

    Query Serving

    • Online DDL Vreplication test suite: adding tests #8213
    • Online DDL/VReplication: more passing tests (no UK/PK) #8225
    • Online DDL: reject ALTER TABLE...RENAME statements #8227
    • Online DDL/VReplication: fail on existence of FOREIGN KEY #8228
    • Online DDL/VReplication: test PK column case change #8336
    • Online DDL/VReplication: add PK DATETIME->TIMESTAMP test #8338
    • Bugfix: assign cexpr.references for column CONVERTed to utf8mb4 #8355

    VReplication

    • Towards a VReplication/OnlineDDL testing suite #8181
    • Online DDL/Vreplication: column type awareness #8239
    • remove duplicate ReadMigrations, fixing build #8258
    • VReplication (and by product, Online DDL): support GENERATED column as part of PRIMARY KEY #8335

    vttestserver

    • docker/vttestserver: Set max_connections in default-fast.cnf at container build time #7810

    Documentation

    Build/CI

    • Update version for latest snapshot #7801
    • v10 GA Release Notes #7964
    • Updates and Corrections to v9 Release Notes #8295

    Query Serving

    • Update link to 'Why FK not supported in Online DDL' blog post #8372

    Other

    • correct several wrong words #7822
    • MAINTAINERS.md: Update enisoc's email. #7909
    • Modification of the Markdown list format in the release notes template #7962
    • Update vreplicator docs #8014
    • Release notes: add Known Issues #8027

    Enhancement

    Observability

    • [trace] Add logging support to opentracing plugins #8289

    Build/CI

    • Makefile: add cross-build target for cross-compiling client binaries #7806
    • git: improve signoff detection #7852
    • Release notes generation #7932
    • vitess/lite docker build fails for mysql80 #7943
    • Fix a gofmt warning #7959
    • docker/vttestserver/run.sh: Add $CHARSET environment variable #7970
    • docker/lite/install_dependencies.sh: If the dependency install loop reaches its max number of retries, consider that a failure; exit the script nonzero so the build halts. #7976
    • Add commit count and authors to the release notes #7982
    • Automate version naming and release tagging #8034
    • Make sure to only allow codegen when the rest of the code compiles #8169
    • Add options to vttestserver to pass -foreign_key_mode, -enable_online_ddl, and -enable_direct_ddl through to vtcombo #8177

    Cluster management

    • Dynamic throttle metric threshold #7742
    • Bump Bootstrap version, per CVE-2018-14040 (orchestrator/vtorc) #7824
    • Check tablet alias before removing after error stream #7915
    • vttablet/tabletmanager - add additional test for tmstate.Open() #7993
    • Add vttablet_restore_done hook #8007
    • Added ValidateVSchema #8012
    • Add RemoteOperationTimeout to both legacy and grpc ChangeTabletType implementations. #8052
    • [vtctldserver] Add guard against self-reparent, plus misc updates #8084
    • [tm_state] updateLocked should re-populate local metadata tables to reflect promotion rule changes #8107
    • [vtctld] Migrate ApplyVSchema to VtctldServer #8113
    • vtctl.generateShardRanges -> key.GenerateShardRanges #8134
    • etcd: add grpc.WithBlock to client config #8205
    • [workflow] Add tracing to GetWorkflows endpoint #8266
    • [vtctldclient] Add legacy shim #8284
    • [vtctldserver] Add tracing #8285
    • [vtctldserver] Add additional backup info fields #8321
    • [workflow] Call scanWorkflow concurrently #8272

    Query Serving

    • [Gen4] Implemented table alias functionality in the semantic analysis #7629
    • Improved error messages (tabletserver) #7747
    • Allow modification of tablet unhealthy_threshold via debugEnv #7753
    • [Gen4] Initial Horizon Planning #7775
    • [tablet, queryrules] Extend query rules to check comments #7784
    • Add support for showing global gtid executed per shard #7856
    • Minor cleanups around errors on the vtgate #7864
    • log unsupported queries #7865
    • Show databases like #7912
    • Add rank as reserved keyword #7944
    • Online DDL: introducing ddl_strategy -singleton-context flag #7946
    • Ignore the error and log as warn if not able to validate the current system setting for check and ignore case #8004
    • Detect and signal schema changes on vttablets #8005
    • DDL bypass plan #8013
    • Online DDL: progress & ETA for Vreplication migrations #8015
    • Scatter errors as warning in olap query #8018
    • livequeryz: livequeryz/terminate link should be relative #8025
    • Handle online DDL user creation if we do not have SUPER privs (e.g. AWS Aurora) #8038
    • Update the schema copy with minimal changes #8067
    • Gen4: Support for order by column not present in projection #8070
    • Schema tracking in vtgate #8074
    • protobuf: upgrade #8075
    • added mysql 8.0 reserved keywords #8086
    • Fix ghost/pt-osc in the external DB case where MySQL might be reporting #8115
    • Support for vtgate -enable_direct_ddl flag #8116
    • add transaction ID to query log for acquisition and analysis #8133
    • Inline reference #8136
    • Improve ScatterErrorsAsWarnings functionality #8139
    • Gen4: planning Select IN #8155
    • Primary key name #8188
    • Online DDL: read and publish gh-ost FATAL message where possible #8192
    • Expose inner net.Conn to be able to write better unit tests #8217
    • vtgate: validate important flags during startup #8218
    • Online DDL/VReplication: AUTO_INCREMENT support and tests #8223
    • [Gen4] some renaming from v4 to Gen4 #8234
    • Schema Tracking: Optimize and Bug Fix #8243
    • gen4: minor refactoring to improve readability #8245
    • gen4: plan more opcodes #8254
    • Online DDL: report rows_copied for migration (initial support in gh-ost) #8255
    • Schema tracking: new tables in sharded keyspace #8256
    • [parser] use table_alias for ENGINE option in CREATE TABLE stmt #8307
    • Add /debug/env for vtgate #8292
    • gen4: outer joins #8312
    • Gen4: expand star in projection list #8325
    • gen4: Fail all queries not handled well by gen4 #8359
    • Gen4 fail more2 #8382

    VReplication

    • Use Dba user when Vexec is runAsAdmin #7731
    • VReplication: add table for logging stream errors, state changes and key workflow steps #7831
    • Fix some static check warning #7960
    • VSchema Validation on ReshardWorkflow Creation #7977
    • VReplication: tracking rows_copied #7980
    • Vdiff formatting improvements #8079
    • VReplication: ignore generated columns in workflows #8129
    • VReplication Copy Phase: Increase default replica lag tolerance. Also make it and copy timeout modifiable via flags #8130
    • Materialize: Add additional comparison operators in Materialize and fix bug where they not applied for sharded keyspaces #8247
    • Copy Phase: turn on OptimizeInserts by default #8248
    • Tracker/VStreamer: only reload schema for tables in current database and not for internal table artifacts #8257
    • Online DDL/Vreplication suite: support ENUM->VARCHAR/TEXT type change #8275
    • Added TableName to DiffReport struct. #8279
    • New VReplication lag metric #8306
    • VStream API: add heartbeat for idle streams #8244
    • Online DDL/VReplication: support non-UTF8 character sets #8322
    • Online DDL/Vreplication suite: fix test for no shared UK #8334
    • Online DDL/VReplication: support DROP+ADD column of same name #8337
    • Online DDL/VReplication test suite: support ENUM as part of PRIMARY KEY #8345

    VTAdmin

    • [vtadmin-web] Add useSyncedURLParam hook to persist filter parameter in the URL #7857
    • [vtadmin-web] Display more data on /gates view and add filtering #7876
    • [vtadmin] Promote ErrNoSchema to a TypedError which returns http 404 #7885
    • [vtadmin] gate/tablet/vtctld FQDNs #7886
    • [vtadmin-web] Display vindex data on Schema view #7917
    • [vtadmin-web] Add filtering and source/target shards to Workflows view #7948
    • [vtadmin-web] Add filtering + shard counts/status to Keyspaces view #7991
    • [vtadmin-web] Display shard state on Tablets view + extract tablet utilities #7999
    • [vtadmin] experimental tabletdebug #8003
    • [vtadmin-web] Display timestamps + stream counts on Workflows view #8009
    • [vtadmin-web] Updates to table styling + other CSS #8072
    • [vtadmin-web] Add Tab components #8119
    • [vtadmin-api] Update GetTablet to use alias instead of hostname #8163
    • [vtadmin-api] Rename flag 'http-tablet-fqdn-tmpl' to 'http-tablet-url-tmpl' + update vtadmin flags for local example #8164
    • Add --tracer flag to vtadmin and actually start tracing #8165
    • [vtadmin-web] The hastiest Tablet view (+ experimental debug vars) #8170
    • [vtadmin-web] Add tabs to Workflow view #8203
    • [vtctld] Add GetSrvVSchemas command #8221
    • [vtadmin-api] Add HTTP endpoints for /api/srvvschemas and /api/srvvschema/{cluster}/{cell} #8226
    • [vtadmin-web] Add QPS and VReplicationQPS charts to Tablet view #8263
    • [vtadmin-web] Add client-side error handling interface + Bugsnag implementation #8287
    • [vtadmin-web] Add chart for stream vreplication lag across all streams in a workflow #8331
    • [vtctldproxy] Add more annotations to vtctld Dial calls #8346

    vttestserver

    • Vttest create db #7989
    • Adds an environment variable to set the MySQL max connections limit in vttestserver docker image #8210

    Feature Request

    Build/CI

    • [trace] Add optional flags to support flexible jaeger configs #8199

    Cluster management

    • Add VtctldServer to vtcombo #7896
    • [vtctldserver] Migrate routing rules RPCs, and also RebuildVSchemaGraph #8197
    • [vtctldserver] Migrate CellInfo, CellAlias rw RPCs #8219
    • [vtctldserver] Add RefreshState RPCs #8232

    Query Serving

    • VTGate grpc implementation of Prepare and CloseSession #8211
    • Schema tracking: One schema load at a time per keyspace #8224

    VTAdmin

    • [vtadmin-web] Add DataFilter + Workspace layout components #8032
    • [vtadmin-web] Add Tooltip + HelpTooltip components #8076
    • [vtadmin-web] Add initial Stream view, render streams on Workflow view #8091
    • [vtadmin-web] The hastiest-ever VTExplain UI #8092
    • [vtadmin-web] Add source-map-explorer util #8093
    • [vtadmin-web] Add Keyspace detail view #8111
    • [vtadmin-api] Add GetKeyspace endpoint #8125
    • [workflow] Add vreplication_log data to workflow protos, and VtctldServer.GetWorkflows method #8261
    • [vtadmin] Add debug endpoints #8268

    Internal Cleanup

    Build/CI

    • Makefile: fix cross-build comments #8246
    • remove unused hooks with refs to master branch #8250
    • [vtctldserver] Update tests to prevent races during tmclient init #8320
    • sks-keyservers.net seems to be finally dead, replace with #8363
    • endtoend: change log level of etcd start from error to info #8370

    Cluster management

    • Online DDL: code cleanup #7589
    • [wrangler|topotools] Migrate UpdateShardRecords, RefreshTabletsByShard, and {Get,Save}RoutingRules #7965
    • Tear down old stream_migrater shim, now that we're fully in package workflow #8073
    • [mysqlctl] Restructure MetadataManager to reduce public API surface area #8152
    • naming: master to primary #8251
    • vtorc: code cleanup #8269

    Query Serving

    • Plan StreamExecute Queries #7941

    VReplication

    • [workflow] extract migration targets from wrangler #7934
    • [wrangler|workflow] Extract vrStream type to workflow.VReplicationStream #7966
    • [wrangler|workflow] Extract workflowState and workflowType out to package workflow #7967
    • [wrangler|workflow] extract *wrangler.streamMigrater to workflow.StreamMigrator #8008
    • [workflow] Migrate getCellsWith{Shard,Table}ReadsSwitched, TrafficSwitchDirection and TableRemovalType to package workflow #8190
    • [workflow] Cleanup wrangler wrappers, migrate checkIfJournalExistsOnTablet to package workflow #8193

    VTAdmin

    • [vtadmin-api] Replace magic numbers with net/http constants #8127
    • [vtadmin-web] Move single-entity view components into subfolders #8202
    • [vtadmin] Ensure we log any errors when closing the tracer #8262

    Other

    Build/CI

    • add vtctldclient to binary directory #7889

    Cluster management

    • vttablet/tabletmanager: add isInSRVKeyspace/isShardServing #7929

    Other

    • Change VitessInputFormat key type #199
    • Online DDL plan via Send; "singleton" migrations on tablets #7785
    • flaky onlineddl tests: reduce -online_ddl_check_interval #7847
    • Looking into flaky endtoend upgrade test #7900
    • fixing flaky upgrade test #7901

    Query Serving

    • Introduce Concatenated Fixed-width Composite or CFC vindex #7537
    • Add common tags for stats backends that support it #7651
    • Add support for showing global vgtid executed #7797
    • perf: vttablet/mysql optimizations #7800
    • Fix bug with reserved connections to stale tablets #7879
    • Memory Sort to close the goroutines when callback returns error #7903
    • OnlineDDL: more migration check ticks upon migration start #7961
    • Update gh-ost binary to v1.1.3 #8021
    • VReplication Online DDL: fix classification of virtual columns #8043

    VReplication

    • VReplicationErrors metric: use . as delimiter instead of _ to behave well with Prometheus #7807

    VTAdmin

    • Update GetSchema filtering to exclude shards where IsMasterServing but no MasterAlias #7805
    • [vtadmin-api] Reintroduce include_non_serving_shards opt to GetSchema #7814
    • [vtadmin-web] Add DataCell component #7817
    • Rewrite useTableDefinitions hook as getTableDefinitions util #7821
    • [vtadmin-web] Display (approximate) table sizes + row count on /schemas view #7826
    • [vtadmin-web] Add Pip + TabletServingPip components #7827

    Performance

    Query Serving

    • vttablet: stream consolidation #7752
    • perf: optimize bind var generation #7828

    VReplication

    • VReplication: optimize the catchup phases by filtering out rows which not within range of copied pks during inserts #7708
    • VReplication: ability to compress gtid when stored in _vt.vreplication's pos column #7877
    • vreplication: performance & benchmarks (table copying) #7881
    • vreplication: dynamic packet sizing #7933
    • perf: vreplication client CPU usage #7951
    • VDIff: fix performance regression introduced by progress logging #8016
    • proto: Generate faster code using vtprotobuf #8173
    • proto: enable pooling for vreplication #8273

    Testing

    Build/CI

    • Attempt to fix TLS Server Flaky test #7842
    • TestMakeCommonTags Flaky Test: match elements #7843
    • upgrade tests: test against v9.0.0 #7848
    • FOSSA scan added #7862
    • ci: fix all racy tests #7904
    • CI: Revert docker change for unit tests from #7868 #7940
    • Add online ddl on start queries to schema list in vtexplain tablet #8397

    Java

    • [Java] JDBC mysql driver test #8154

    Other

    • Fuzzing: Fixup oss-fuzz build script #7782
    • Wrangler tests: Return a fake tablet in the wrangler test dialer to avoid tablet picker errors spamming the test logs #7863

    Query Serving

    • clean up test #7816
    • tabletserver: fix flaky test #7851
    • Planbuilder: Add fuzzer #7902
    • mysql: Small adjustments to fuzzer #7907
    • vtgate/engine: Add fuzzer #7914
    • Adding Fuzzer Test Cases #8106
    • Addition of fuzzer issues #8195

    VReplication

    • vstreamer: Add fuzzer #7918

    vttestserver

    • Speedup new vttestserver tests #8229
    • Vttestserver docker test #8253

    Cluster management

    • Make timestamp authoritative for master information #8381

    The release includes 1041 commits (excluding merges)

    Thanks to all our contributors: @AdamKorcz, @GuptaManan100, @Hellcatlk, @Johnny-Three, @acharisshopify, @ajm188, @alexrs, @aquarapid, @askdba, @deepthi, @dependabot[bot], @doeg, @dyv, @enisoc, @frouioui, @gedgar, @guidoiaquinti, @harshit-gangal, @hkdsun, @idvoretskyi, @kirs, @mcronce, @narcsfz, @noxiouz, @rafael, @rohit-nayak-ps, @setassociative, @shlomi-noach, @systay, @tokikanno, @vmg, @wangmeng99, @yangxuanjia, @zhangshj-inspur

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.0-rc1-740c379.tar.gz(396.98 MB)
    vitess-11.0.0_rc1-740c379.x86_64.rpm(394.71 MB)
    vitess_11.0.0-rc1-740c379_amd64.deb(396.99 MB)
  • v10.0.2(May 27, 2021)

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v10.0.2, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v10.0.5 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Bug fixes

    Query Serving

    • Fixes encoding of sql strings #8029
    • Fix for issue with information_schema queries with both table name and schema name predicates #8099
    • PRIMARY in index hint list for release 10.0 #8159

    VReplication

    • VReplication: Pad binlog values for binary() columns to match the value returned by mysql selects #8137

    CI/Build

    Build/CI

    • update release notes with known issue #8081

    Documentation

    Other

    • Post v10.0.1 updates #8045

    Enhancement

    Build/CI

    • Added release script to the makefile #8030

    Other

    • Add optional TLS feature to gRPC servers #8176

    Other

    Build/CI

    • Release 10.0.1 #8031

    The release includes 14 commits (excluding merges) Thanks to all our contributors: @GuptaManan100, @askdba, @deepthi, @harshit-gangal, @noxiouz, @rohit-nayak-ps, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.2-fc78470.tar.gz(386.97 MB)
    vitess-10.0.2-fc78470.x86_64.rpm(384.69 MB)
    vitess_10.0.2-fc78470_amd64.deb(386.96 MB)
  • v9.0.2(May 25, 2021)

    Bug fixes

    Cluster management

    • Restore: Check disable_active_reparents properly before waiting for position update #8114

    Query Serving

    • Fix information_schema query with system schema in table_schema filter #8095
    • Fix for issue with information_schema queries with both table name and schema name predicates #8096
    • Fix for transactions not allowed to finish during PlannedReparentShard #8098
    • PRIMARY in index hint list #8158

    CI/Build

    Build/CI

    • Release 9.0.1 #8065
    • 9.0.0: update release notes with known issue #8080 #8082
    • Added release script to the makefile #8182
    • Update do_release to work in the 9.0 branch #8184

    Performance

    Cluster management

    • Revert "backup: Use pargzip instead of pgzip for compression." #8174

    The release includes 17 commits (excluding merges) Thanks to all our contributors: @GuptaManan100, @deepthi, @Harshit, @rafael, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-9.0.2-58cd7b0.tar.gz(318.84 MB)
    vitess-9.0.2-58cd7b0.x86_64.rpm(316.98 MB)
    vitess_9.0.2-58cd7b0_amd64.deb(318.85 MB)
  • v9.0.1(May 6, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Bug fixes

    Cluster management

    • Backport: Respect -disable_active_reparents in backup/restore #8063

    Other

    • [9.0] fix regression - should be able to plan subquery on top of subquery #7683

    Query Serving

    • [9.0] Fix for reserved connection usage with transaction #7666
    • [9.0] Fix MySQL Workbench failure on login with select current_user() #7706
    • [9.0] make sure to handle subqueries on top of subqueries #7776
    • [9.0] make sure to not log sensitive information #7778
    • [9.0] ddl bypass planner #8035
    • [9.0] Memory Sort to close the goroutines when callback returns error #8040
    • [9.0] Fix bug with reserved connections to stale tablets #8041
    • [9.0] Fix for Query Serving when Toposerver is Down #8046
    • ignore the error and log as warn if not able to validate the current system settings value #8062

    VReplication

    • VReplication: fix vreplication timing metrics #8060
    • VReplication: Pad binlog values for binary() columns to match the value returned by mysql selects #8061

    Documentation

    Other

    • 9.0.0 Release Notes #7384

    Other

    Build/CI

    • Fix Dockerfiles for vtexplain and vtctlclient #7418

    Query Serving

    • Fix table parsing on VSchema generation #7511
    • [9.0] Show anywhere plan fix to consider default keyspace #7530
    • [9.0] Reset Session for Reserved Connection when the connection id is not found #7544
    • Healthcheck: update healthy tablets correctly when a stream returns an error or times out #7732

    VReplication

    • MoveTables: Refresh SrvVSchema and source tablets on completion #8059

    The release includes 43 commits (excluding merges) Thanks to all our contributors: @askdba, @deepthi, @dyv, @harshit-gangal, @rafael, @rohit-nayak-ps, @shlomi-noach, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-9.0.1-42c38e5.tar.gz(318.28 MB)
    vitess-9.0.1-42c38e5.x86_64.rpm(316.42 MB)
    vitess_9.0.1-42c38e5_amd64.deb(318.28 MB)
  • v10.0.1(May 4, 2021)

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v10.0.1, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v10.0.5 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Bugs Fixed

    • Running binaries with --version or calling @@version from a MySQL client still shows 10.0.0-RC1
    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.1-f7304cd.tar.gz(386.89 MB)
    vitess-10.0.1-f7304cd.x86_64.rpm(384.60 MB)
    vitess_10.0.1-f7304cd_amd64.deb(386.89 MB)
  • v10.0.0(Apr 27, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Known Issues

    • Running binaries with --version or running select @@version from a MySQL client still shows 10.0.0-RC1

    • Online DDL cannot be used if you are using the keyspace filtering feature of VTGate

    • VReplication errors when a fixed-length binary column is used as the sharding key #8080

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v10.0.0, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v10.0.5 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Bugs Fixed

    VTGate / MySQL compatibility

    • Remove printing of ENFORCED word so that statements remain compatible with mysql 5.7 #7458
    • Allow any ordering of generic options in column definitions #7459
    • Corrects the comment handling in vitess #7581
    • Fix regression - should be able to plan subquery on top of subquery #7682
    • Nullable Timestamp Column Fix #7740
    • VTGate: Fix the error messages in drop, create and alter database commands #7397
    • VTGate: Fix information_schema query with system schema in table_schema filter #7430
    • VTGate: Fix Set Statement in Tablet when executed with bindvars #7431
    • VTGate: Fix for Query Serving when Toposerver is Down #7484
    • VTGate: Add necessary bindvars when preparing queries #7493
    • VTGate: Show anywhere plan fix to consider default keyspace #7531
    • VTGate: Fix table parsing on VSchema generation #7511
    • VTGate: Use the emulated MySQL version for special comments #7510
    • VTGate: Reset Session for Reserved Connection when the connection id is not found #7539
    • VTGate: Healthcheck: update healthy tablets correctly when a stream returns an error or timeout #7732
    • VTGate: Fix for reserved connection usage with transaction #7646
    • VTGate: Fix MySQL Workbench failure on login with select current_user() #7705
    • VTGate: Constraint names and database names with spaces. #7745
    • VTGate: Fix dual table query when system schema is selected database #7734

    Other

    • VTTablet: Correctly initialize statsTabletTypeCounts during startup #7390
    • Backup/Restore: Respect -disable_active_reparents in backup/restore #7576
    • Backup/Restore: check disable_active_reparents properly before waiting for position update #7703

    Functionality Added or Changed

    VTGate / MySQL compatibility / Query Serving

    • VTGate: Gen4 Planner: AxB vs BxA #7274
    • VTGAte: Gen4 fallback planning #7370
    • VTGate: Support for CALL procedures #7287
    • VTGate: Set default for @@workload to OLTP #7288
    • VTGate: Change @@version and @@version_comment #7337
    • VTGate: Fix VitessAware system variables of type boolean return NULL when MySQL is not involved #7353
    • VTGate: Add stats for RowsAffected similar to RowsReturned #7380
    • VTGate: Added information_schema_stats_expiry to allowed list of set vars #7435
    • VTGate: LFU Cache Implementation #7439
    • VTGate: Describe table to route based on table name and qualifier #7445
    • VTGate: Olap error message fix #7448
    • VTGate: Temporary Table support in unsharded keyspace #7411
    • VTGate: Publish table size on schema #7444
    • VTGate: Support for caching_sha2_password plugin in mysql/client #6716
    • VTGate: Moving Show plan from executor to planbuilder #7475
    • VTGate: Adds another case to merge routes for information_schema queries #7504
    • VTGate: Add innodb_read_rows as vttablet metric #7520
    • VTGate: Adds support for Show variables #7547
    • VTGate: gen4: fail unsupported queries #7409
    • VTGate: Fix Metadata in SHOW queries #7540
    • VTGate: Update AST helper generation #7558
    • VTGate: Avoiding addition of redundant unary operators #7579
    • VTGate: Optimise AST rewriting #7583
    • VTGate: Add Show Status query to vtexplain and make asthelpergen/sizegen quiet #7590
    • VTGate: Add support for SELECT ALL #7593
    • VTGate: Empty statement error code change in sql parsing #7618
    • VTGate: Socket system variable to return vitess mysql socket #7637
    • VTGate: Make DROP/CREATE DATABASE pluggable #7381
    • VTGate: Allow Select with lock to pass through in vttablet #7584
    • VTGate: Fix ordering in SELECT INTO and printing of strings #7655
    • VTGate: AST Equals code generator #7672
    • VTGate: [tabletserver] More resilient wait for schema changes #7684
    • VTGate: Fix flush statement planner #7695
    • VTGate: Produce query warnings for using features not supported when sharded #7538
    • VTGate: Support for ALTER VITESS_MIGRATION statements #7663
    • VTGate: Solve I_S queries using CNF rewriting #7677
    • VTGate: System schema queries #7685
    • VTGate: Make the AST visitor faster #7701
    • VTGate: COM_PREPARE - Single TCP response packet with all MySQL Packets #7713
    • VTGate: Replace the database name in result fields only if needed #7714
    • VTGate: Split ast_helper into individual gen files #7727
    • VTGate: Adds support for ordering on character fields for sharded keyspace queries #7678
    • VTGate: Show columns query on system schema #7729
    • VTGate: Disallow foreign key constraint on ddl #7780
    • VTGate: VTGate: support -enable_online_ddl flag #7694
    • VTGate: Default to false for system settings to be changed per session at the database connection level #7921
    • VTGate: vtctl: return error on invalid ddl_strategy #7924
    • VTGate: [10.0] Squashed backport of #7903 #7927
    • VTGate: [10.0] Fix bug with reserved connections to stale tablets #7935
    • VTGate: [10.0] Fix for keyspaces_to_watch regression #7936
    • VTGate: [10.0] Update healthy tablets correctly for primary down #7937
    • VTGate: [10.0] Allow modification of tablet unhealthy_threshold via debugEnv #7938

    Testing

    • Fuzzing: Add vtctl fuzzer #7605
    • Fuzzing: Add more fuzzers #7622
    • Fuzzing: Add 3 fuzzers for mysql endpoints #7639
    • Fuzzing: Add oss-fuzz build script #7591
    • Fuzzing: Add requirement for minimum length of input data #7722
    • Fuzzing: Add new mysql fuzzer #7660
    • Fuzzing: Add [grpcvtgateconn] fuzzer #7689
    • Fuzzing: Make mysql fuzzer more calls during each iteration #7766

    Performance

    • VTGate: [perf] zero-copy tokenizer #7619
    • VTGate: [perf: sqlparser faster formatting #7710
    • VTGate :[perf] Cache reserved bind variables in queries #7698
    • VTGate: [perf] sqlparser yacc codegen #7669
    • VTGate: Making fast AST rewriter faster #7726
    • VTGate: Cached Size Implementation #7387
    • VTGate: Plan remove mutexes #7468
    • LFU Cache Bug Fixes #7479
    • [cache] Handle all possible initialization cases #7556
    • VTGate: [servenv] provide a global flag for profiling #7496
    • VTGate: [vttablet] Benchmarks and two small optimizations #7560
    • [pprof]: allow stopping profiling early with a signal #7594
    • perf: RPC Serialization #7519
    • perf: keyword lookups in the tokenizer #7606

    Cluster Management

    • [vtctld] Migrate topo management RPCs #7395
    • [vtctldclient] Set SilenceErrors on the root command, so we don't double-log #7404
    • [vtctldclient] Command line flags: dashes and underscores synonyms #7406
    • Extract the maxReplPosSearch struct out to topotools #7420
    • Add protoutil package, refactor ISP to use it #7421
    • Add ErrorGroup to package concurrency, use in waitOnNMinusOneTablets #7429
    • [vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer #7434
    • [vtctld/wrangler] Extract EmergencyReparentShard logic to dedicated struct and add unit tests #7464
    • Provide named function for squashing usage errors; start using it #7451
    • [concurrency] Add guard against potentially blocking forever in ErrorGroup.Wait() when NumGoroutines is 0 #7463
    • Add hook for statsd integration #7417
    • [concurrency] Add guard against potentially blocking forever in ErrorGroup.Wait() when NumGoroutines is 0 #7463
    • Resilient rebuild keyspace graph check, tablet controls not in RebuildKeyspaceGraph command #7442
    • [reparentutil / ERS] confirm at least one replica succeeded to SetMaster, or fail #7486
    • [reparentutil / wrangler] Extract PlannedReparentShard logic from wrangler to PlannedReparenter struct #7501
    • Add backup/restore duration stats #7512
    • Refresh replicas and rdonly after MigrateServedTypes except on skipRefreshState. #7327
    • [eparentutil] ERS should not attempt to WaitForRelayLogsToApply on primary tablets that were not running replication #7523
    • [orchestrator] prevent XSS attack via 'orchestrator-msg' params #7526
    • [vtctld] Add remaining reparent commands to VtctldServer #7536
    • [reparentutil] ERS should not attempt to WaitForRelayLogsToApply on primary tablets that were not running replication #7523
    • Shutdown vttablet gracefully if tablet record disappears #7563
    • ApplySchema: -skip_preflight #7587
    • Table GC: disable binary logging on best effort basis #7588
    • Addition of waitSig pprof argument to start recording on USR1 #7616
    • Add combine TLS certs feature #7609
    • Check error response before attempting to access InitShardPrimary response #7658
    • [vtctld] Migrate GetSrvKeyspace as GetSrvKeyspaces in VtctldServer #7680
    • [vtctld] Migrate ShardReplicationPositions #7690
    • [reparentutil | ERS] Bind status variable to each goroutine in WaitForRelayLogsToApply #7692
    • [servenv] Fix var shadowing caused by short variable declaration #7702
    • [vtctl|vtctldserver] List/Get Tablets timeouts #7715
    • vtctl ApplySchema supports '-request_context' flag #7777

    VReplication

    • VReplication: vstreamer to throttle on source endpoint #7324
    • VReplication: Throttle on target tablet #7364
    • VReplication: Throttler: fix to client usage in vreplication and table GC #7422
    • VReplication: MoveTables/Reshard add flags to start workflows in a stopped state and to stop workflow once copy is completed #7449
    • VReplication: Support for caching_sha2_password plugin in mysql/client #6716
    • VReplication: Validate SrvKeyspace during Reshard/SwitchReads #7481
    • VReplication: [MoveTables] Refresh SrvVSchema (for Routing Rules) and source tablets (for Blacklisted Tables) on completion #7505
    • VReplication : Data migration from another Vitess cluster #7546
    • VReplication : [materialize] Add cells and tablet_types parameters #7562
    • VReplication: JSON Columns: fix bug where vreplication of update statements were failing #7640
    • VReplication: Make the frequency at which heartbeats update the _vt.vreplication table configurable #7659
    • VReplication: Error out if binlog compression is turned on #7670
    • VReplication: Tablet throttler: support for custom query & threshold #7541
    • VStream API: allow aligning streams from different shards to minimize skews across the streams #7626
    • VReplication: Backport 7809: Update rowlog for the API change made for the vstream skew alignment feature #7890

    OnlineDDL

    • OnlineDDL: update gh-ost binary to v1.1.1 #7394
    • Online DDL via VReplication #7419
    • Online DDL: VReplicatoin based mini stress test CI #7492
    • OnlineDDL: Revert for VReplication based migrations #7478
    • Online DDL: Internal support for eta_seconds #7630
    • Online DDL: Support 'SHOW VITESS_MIGRATIONS' queries #7638
    • Online DDL: Support for REVERT VITESS_MIGRATION statement #7656
    • Online DDL: Declarative Online DDL #7725

    VTAdmin

    Vitess 10.0 introduces a highly-experimental multi-cluster admin API and web UI, called VTAdmin. Deploying the vtadmin-api and vtadmin-web components is completely opt-in. If you're interested in trying it out and providing early feedback, come find us in #feat-vtadmin in the Vitess slack. Note that VTAdmin relies on the new VtctldServer API, so you must be running the new grpc-vtctld service on your vtctlds in order to use it.

    • VTAdmin: Add vtadmin-web build flag for configuring fetch credentials #7414
    • VTAdmin: Add cluster field to vtadmin-api's /api/gates response #7425
    • VTAdmin: Add /api/clusters endpoint to vtadmin-api #7426
    • VTAdmin: Add /api/schemas endpoint to vtadmin-api #7432
    • VTAdmin: [vtadmin-web] Add routes and simple tables for all entities #7440
    • VTAdmin: [vtadmin-web] Set document.title from route components #7450
    • VTAdmin: [vtadmin-web] Add DataTable component with URL pagination #7487
    • VTAdmin: [vtadmin-api] Add shards to /api/keyspaces response #7453
    • VTAdmin: [vtadmin-web] Add replaceQuery + pushQuery to useURLQuery hook #7507
    • VTAdmin: [vtadmin-web] An initial pass for tablet filters #7515
    • VTAdmin: [vtadmin-web] Add a Select component #7524
    • VTAdmin: [vtadmin-api] Add /vtexplain endpoint #7528
    • VTAdmin: [vtadmin-api] Reorganize tablet-related functions into vtadmin/cluster/cluster.go #7553
    • VTAdmin: Three small bugfixes in Tablets table around stable sort order, display type lookup, and filtering by type #7568
    • VTAdmin: [vtadmin] Add GetSchema endpoint #7596
    • VTAdmin: [vtadmin/testutil] Add testutil helper to manage the complexity of recursively calling WithTestServer #7601
    • VTAdmin: [vtadmin] Add FindSchema route #7610
    • VTAdmin: [vtadmin-web] Add simple /schema view with table definition #7615
    • VTAdmin: [vtadmin] vschemas api endpoints #7625
    • VTAdmin: [vtadmin] Add support for additional service healthchecks in grpcserver #7635
    • VTAdmin: [vtadmin] test refactors #7641
    • VTAdmin: [vtadmin] propagate error contexts #7642
    • VTAdmin: [vtadmin] tracing refactor #7649
    • VTAdmin: [vtadmin] GetWorkflow(s) endpoints #7662
    • VTAdmin: [vitessdriver|vtadmin] Support Ping in vitessdriver, use in vtadmin to healthcheck connections during Dial #7709
    • VTAdmin: [vtadmin] Add to local example #7699
    • VTAdmin: [vtexplain] lock #7724
    • VTAdmin: [vtadmin] Aggregate schema sizes #7751
    • VTAdmin: [vtadmin-web] Add comments + 'options' parameter to API hooks #7754
    • VTAdmin: [vtadmin-web] Add common max-width to infrastructure table views #7760
    • VTAdmin: [vtadmin-web] Add hooks + skeleton view for workflows #7762
    • VTAdmin: [vtadmin-web] Add a hasty filter input to the /schemas view #7779

    Other / Tools

    • [rulesctl] Implements CLI tool for rule management #7712

    Examples / Tutorials

    • Source correct shell script in README #7749

    Documentation

    • Add Severity Labels document #7542
    • Remove Google Groups references #7664
    • Move some commas around in README.md :) #7671
    • Add Andrew Mason to Maintainers List #7757

    Build/CI Environment Changes

    • Update java build versions to vitess 10.0.0 #7383
    • CI: check run analysis to post JSON from file #7386
    • Fix Dockerfiles for vtexplain and vtctlclient #7418
    • CI: Add descriptive names to vrep shards. Update test generator script #7454
    • CI: adding 'go mod tidy' test #7461
    • Docker builds vitess/vtctlclient to install curl #7466
    • Add VT_BASE_VER to vtexplain/Dockerfile #7467
    • Enable -mysql_server_version in vttestserver, and utilize it in vttestserver container images #7474
    • [vtctld | tests only] testtmclient refactor #7518
    • CI: skip some tests on forked repos #7527
    • Workflow to check make sizegen #7535
    • Add mysqlctl docker image #7557
    • Restore CI workflow shard 26, accidentally dropped #7569
    • Update CODEOWNERS #7586
    • CI: ci-workflow-gen turn string to array to reduce conflicts #7582
    • Add percona-toolkit (for pt-osc/pt-online-schema-change) to the docker/lite images #7603
    • CI: Use ubuntu-18.04 in tests #7614
    • [vttestserver] Fix to work with sharded keyspaces #7617
    • Add tools.go #7517
    • Make vttestserver compatible with persistent data directories #7718
    • Add vtorc binary for rpm,deb builds #7750
    • Fixes bug that prevents creation of logs directory #7761
    • [Java] Guava update to 31.1.1 #7764
    • make: build vitess as static binaries by default #7795 ← Potentially breaking change
    • make: build vitess as static binaries by default (10.0 backport) #7808
    • java: prepare java version for release 10.0 #7922

    Functionality Neutral Changes

    • VTGate: Remove unused key.Destination.IsUnique() #7565
    • VTGate: Add information_schema query on prepare statement #7746
    • VTGate: Tests for numeric_precision and numeric_scale columns in information_schema #7763
    • Disable flaky test until it can be fixed #7623
    • Tests: reset stat at the beginning of test #7644
    • Cleanup mysql server_test #7645
    • vttablet: fix flaky tests #7543
    • Removed unused tests for Wordpress installation #7516
    • Fix unit test fail after merge #7550
    • Add test with NULL input values for vindexes that did not have any. #7552

    VtctldServer

    As part of an ongoing effort to transition from the VtctlServer gRPC API to the newer VtctldServer gRPC API, we have updated the local example to use the corresponding new vtctldclient to perform InitShardPrimary (formerly, InitShardMaster) operations.

    To enable the new VtctldServer in your vtctld components, update the -service_map flag to include grpc-vtctld. You may specify both grpc-vtctl,grpc-vtctld to gracefully transition.

    The migration is still underway, but you may begin to transition to the new client for migrated commands. For a full listing, refer either to proto/vtctlservice.proto or consult vtctldclient --help.

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.0-48dccf5.tar.gz(386.88 MB)
    vitess-10.0.0-48dccf5.x86_64.rpm(384.60 MB)
    vitess_10.0.0-48dccf5_amd64.deb(386.89 MB)
  • v10.0.0-rc1(Apr 6, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    The following PRs made changes to behaviors that clients might rely on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.

    Bugs Fixed

    VTGate / MySQL compatibility

    • Remove printing of ENFORCED word so that statements remain compatible with mysql 5.7 #7458
    • Allow any ordering of generic options in column definitions #7459
    • Corrects the comment handling in vitess #7581
    • Fix regression - should be able to plan subquery on top of subquery #7682
    • Nullable Timestamp Column Fix #7740
    • VTGate: Fix the error messages in drop, create and alter database commands #7397
    • VTGate: Fix information_schema query with system schema in table_schema filter #7430
    • VTGate: Fix Set Statement in Tablet when executed with bindvars #7431
    • VTGate: Fix for Query Serving when Toposerver is Down #7484
    • VTGate: Add necessary bindvars when preparing queries #7493
    • VTGate: Show anywhere plan fix to consider default keyspace #7531
    • VTGate: Fix table parsing on VSchema generation #7511
    • VTGate: Use the emulated MySQL version for special comments #7510
    • VTGate: Reset Session for Reserved Connection when the connection id is not found #7539
    • VTGate: Healthcheck: update healthy tablets correctly when a stream returns an error or timeout #7732
    • VTGate: Fix for reserved connection usage with transaction #7646
    • VTGate: Fix MySQL Workbench failure on login with select current_user() #7705
    • VTGate: Constraint names and database names with spaces. #7745
    • VTGate: Fix dual table query when system schema is selected database #7734

    Other

    • VTTablet: Correctly initialize statsTabletTypeCounts during startup #7390
    • Backup/Restore: Respect -disable_active_reparents in backup/restore #7576
    • Backup/Restore: check disable_active_reparents properly before waiting for position update #7703

    Functionality Added or Changed

    VTGate / MySQL compatibility / Query Serving

    • VTGate: Gen4 Planner: AxB vs BxA #7274
    • VTGAte: Gen4 fallback planning #7370
    • VTGate: Support for CALL procedures #7287
    • VTGate: Set default for @@workload to OLTP #7288
    • VTGate: Change @@version and @@version_comment #7337
    • VTGate: Fix VitessAware system variables of type boolean return NULL when MySQL is not involved #7353
    • VTGate: Add stats for RowsAffected similar to RowsReturned #7380
    • VTGate: Added information_schema_stats_expiry to allowed list of set vars #7435
    • VTGate: LFU Cache Implementation #7439
    • VTGate: Describe table to route based on table name and qualifier #7445
    • VTGate: Olap error message fix #7448
    • VTGate: Temporary Table support in unsharded keyspace #7411
    • VTGate: Publish table size on schema #7444
    • VTGate: Support for caching_sha2_password plugin in mysql/client #6716
    • VTGate: Moving Show plan from executor to planbuilder #7475
    • VTGate: Adds another case to merge routes for information_schema queries #7504
    • VTGate: Add innodb_read_rows as vttablet metric #7520
    • VTGate: Adds support for Show variables #7547
    • VTGate: gen4: fail unsupported queries #7409
    • VTGate: Fix Metadata in SHOW queries #7540
    • VTGate: Update AST helper generation #7558
    • VTGate: Avoiding addition of redundant unary operators #7579
    • VTGate: Optimise AST rewriting #7583
    • VTGate: Add Show Status query to vtexplain and make asthelpergen/sizegen quiet #7590
    • VTGate: Add support for SELECT ALL #7593
    • VTGate: Empty statement error code change in sql parsing #7618
    • VTGate: Socket system variable to return vitess mysql socket #7637
    • VTGate: Make DROP/CREATE DATABASE pluggable #7381
    • VTGate: Allow Select with lock to pass through in vttablet #7584
    • VTGate: Fix ordering in SELECT INTO and printing of strings #7655
    • VTGate: AST Equals code generator #7672
    • VTGate: [tabletserver] More resilient wait for schema changes #7684
    • VTGate: Fix flush statement planner #7695
    • VTGate: Produce query warnings for using features not supported when sharded #7538
    • VTGate: Support for ALTER VITESS_MIGRATION statements #7663
    • VTGate: Solve I_S queries using CNF rewriting #7677
    • VTGate: System schema queries #7685
    • VTGate: Make the AST visitor faster #7701
    • VTGate: COM_PREPARE - Single TCP response packet with all MySQL Packets #7713
    • VTGate: Replace the database name in result fields only if needed #7714
    • VTGate: Split ast_helper into individual gen files #7727
    • VTGate: Adds support for ordering on character fields for sharded keyspace queries #7678
    • VTGate: Show columns query on system schema #7729
    • VTGate: Disallow foreign key constraint on ddl #7780
    • VTGate: VTGate: support -enable_online_ddl flag #7694

    Testing

    • Fuzzing: Add vtctl fuzzer #7605
    • Fuzzing: Add more fuzzers #7622
    • Fuzzing: Add 3 fuzzers for mysql endpoints #7639
    • Fuzzing: Add oss-fuzz build script #7591
    • Fuzzing: Add requirement for minimum length of input data #7722
    • Fuzzing: Add new mysql fuzzer #7660
    • Fuzzing: Add [grpcvtgateconn] fuzzer #7689
    • Fuzzing: Make mysql fuzzer more calls during each iteration #7766

    Performance

    • VTGate: [perf] zero-copy tokenizer #7619
    • VTGate: [perf: sqlparser faster formatting #7710
    • VTGate :[perf] Cache reserved bind variables in queries #7698
    • VTGate: [perf] sqlparser yacc codegen #7669
    • VTGate: Making fast AST rewriter faster #7726
    • VTGate: Cached Size Implementation #7387
    • VTGate: Plan remove mutexes #7468
    • LFU Cache Bug Fixes #7479
    • [cache] Handle all possible initialization cases #7556
    • VTGate: [servenv] provide a global flag for profiling #7496
    • VTGate: [vttablet] Benchmarks and two small optimizations #7560
    • [pprof]: allow stopping profiling early with a signal #7594
    • perf: RPC Serialization #7519
    • perf: keyword lookups in the tokenizer #7606

    Cluster Management

    • [vtctld] Migrate topo management RPCs #7395
    • [vtctldclient] Set SilenceErrors on the root command, so we don't double-log #7404
    • [vtctldclient] Command line flags: dashes and underscores synonyms #7406
    • Extract the maxReplPosSearch struct out to topotools #7420
    • Add protoutil package, refactor ISP to use it #7421
    • Add ErrorGroup to package concurrency, use in waitOnNMinusOneTablets #7429
    • [vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer #7434
    • [vtctld/wrangler] Extract EmergencyReparentShard logic to dedicated struct and add unit tests #7464
    • Provide named function for squashing usage errors; start using it #7451
    • [concurrency] Add guard against potentially blocking forever in ErrorGroup.Wait() when NumGoroutines is 0 #7463
    • Add hook for statsd integration #7417
    • [concurrency] Add guard against potentially blocking forever in ErrorGroup.Wait() when NumGoroutines is 0 #7463
    • Resilient rebuild keyspace graph check, tablet controls not in RebuildKeyspaceGraph command #7442
    • [reparentutil / ERS] confirm at least one replica succeeded to SetMaster, or fail #7486
    • [reparentutil / wrangler] Extract PlannedReparentShard logic from wrangler to PlannedReparenter struct #7501
    • Add backup/restore duration stats #7512
    • Refresh replicas and rdonly after MigrateServedTypes except on skipRefreshState. #7327
    • [eparentutil] ERS should not attempt to WaitForRelayLogsToApply on primary tablets that were not running replication #7523
    • [orchestrator] prevent XSS attack via 'orchestrator-msg' params #7526
    • [vtctld] Add remaining reparent commands to VtctldServer #7536
    • [reparentutil] ERS should not attempt to WaitForRelayLogsToApply on primary tablets that were not running replication #7523
    • Shutdown vttablet gracefully if tablet record disappears #7563
    • ApplySchema: -skip_preflight #7587
    • Table GC: disable binary logging on best effort basis #7588
    • Addition of waitSig pprof argument to start recording on USR1 #7616
    • Add combine TLS certs feature #7609
    • Check error response before attempting to access InitShardPrimary response #7658
    • [vtctld] Migrate GetSrvKeyspace as GetSrvKeyspaces in VtctldServer #7680
    • [vtctld] Migrate ShardReplicationPositions #7690
    • [reparentutil | ERS] Bind status variable to each goroutine in WaitForRelayLogsToApply #7692
    • [servenv] Fix var shadowing caused by short variable declaration #7702
    • [vtctl|vtctldserver] List/Get Tablets timeouts #7715
    • vtctl ApplySchema supports '-request_context' flag #7777

    VReplication

    • VReplication: vstreamer to throttle on source endpoint #7324
    • VReplication: Throttle on target tablet #7364
    • VReplication: Throttler: fix to client usage in vreplication and table GC #7422
    • VReplication: MoveTables/Reshard add flags to start workflows in a stopped state and to stop workflow once copy is completed #7449
    • VReplication: Support for caching_sha2_password plugin in mysql/client #6716
    • VReplication: Validate SrvKeyspace during Reshard/SwitchReads #7481
    • VReplication: [MoveTables] Refresh SrvVSchema (for Routing Rules) and source tablets (for Blacklisted Tables) on completion #7505
    • VReplication : Data migration from another Vitess cluster #7546
    • VReplication : [materialize] Add cells and tablet_types parameters #7562
    • VReplication: JSON Columns: fix bug where vreplication of update statements were failing #7640
    • VReplication: Make the frequency at which heartbeats update the _vt.vreplication table configurable #7659
    • VReplication: Error out if binlog compression is turned on #7670
    • VReplication: Tablet throttler: support for custom query & threshold #7541
    • VStream API: allow aligning streams from different shards to minimize skews across the streams #7626

    OnlineDDL

    • OnlineDDL: update gh-ost binary to v1.1.1 #7394
    • Online DDL via VReplication #7419
    • Online DDL: VReplicatoin based mini stress test CI #7492
    • OnlineDDL: Revert for VReplication based migrations #7478
    • Online DDL: Internal support for eta_seconds #7630
    • Online DDL: Support 'SHOW VITESS_MIGRATIONS' queries #7638
    • Online DDL: Support for REVERT VITESS_MIGRATION statement #7656
    • Online DDL: Declarative Online DDL #7725

    VTAdmin

    • VTAdmin: Add vtadmin-web build flag for configuring fetch credentials #7414
    • VTAdmin: Add cluster field to vtadmin-api's /api/gates response #7425
    • VTAdmin: Add /api/clusters endpoint to vtadmin-api #7426
    • VTAdmin: Add /api/schemas endpoint to vtadmin-api #7432
    • VTAdmin: [vtadmin-web] Add routes and simple tables for all entities #7440
    • VTAdmin: [vtadmin-web] Set document.title from route components #7450
    • VTAdmin: [vtadmin-web] Add DataTable component with URL pagination #7487
    • VTAdmin: [vtadmin-api] Add shards to /api/keyspaces response #7453
    • VTAdmin: [vtadmin-web] Add replaceQuery + pushQuery to useURLQuery hook #7507
    • VTAdmin: [vtadmin-web] An initial pass for tablet filters #7515
    • VTAdmin: [vtadmin-web] Add a Select component #7524
    • VTAdmin: [vtadmin-api] Add /vtexplain endpoint #7528
    • VTAdmin: [vtadmin-api] Reorganize tablet-related functions into vtadmin/cluster/cluster.go #7553
    • VTAdmin: Three small bugfixes in Tablets table around stable sort order, display type lookup, and filtering by type #7568
    • VTAdmin: [vtadmin] Add GetSchema endpoint #7596
    • VTAdmin: [vtadmin/testutil] Add testutil helper to manage the complexity of recursively calling WithTestServer #7601
    • VTAdmin: [vtadmin] Add FindSchema route #7610
    • VTAdmin: [vtadmin-web] Add simple /schema view with table definition #7615
    • VTAdmin: [vtadmin] vschemas api endpoints #7625
    • VTAdmin: [vtadmin] Add support for additional service healthchecks in grpcserver #7635
    • VTAdmin: [vtadmin] test refactors #7641
    • VTAdmin: [vtadmin] propagate error contexts #7642
    • VTAdmin: [vtadmin] tracing refactor #7649
    • VTAdmin: [vtadmin] GetWorkflow(s) endpoints #7662
    • VTAdmin: [vitessdriver|vtadmin] Support Ping in vitessdriver, use in vtadmin to healthcheck connections during Dial #7709
    • VTAdmin: [vtadmin] Add to local example #7699
    • VTAdmin: vtexplain lock #7724
    • VTAdmin: [vtadmin] Aggregate schema sizes #7751
    • VTAdmin: [vtadmin-web] Add comments + 'options' parameter to API hooks #7754
    • VTAdmin: [vtadmin-web] Add common max-width to infrastructure table views #7760
    • VTAdmin: [vtadmin-web] Add hooks + skeleton view for workflows #7762
    • VTAdmin: [vtadmin-web] Add a hasty filter input to the /schemas view #7779

    Other / Tools

    • [rulesctl] Implements CLI tool for rule management #7712

    Examples / Tutorials

    • Source correct shell script in README #7749

    Documentation

    • Add Severity Labels document #7542
    • Remove Google Groups references #7664
    • Move some commas around in README.md :) #7671
    • Add Andrew Mason to Maintainers List #7757

    Build/CI Environment Changes

    • Update java build versions to vitess 10.0.0 #7383
    • CI: check run analysis to post JSON from file #7386
    • Fix Dockerfiles for vtexplain and vtctlclient #7418
    • CI: Add descriptive names to vrep shards. Update test generator script #7454
    • CI: adding 'go mod tidy' test #7461
    • Docker builds vitess/vtctlclient to install curl #7466
    • Add VT_BASE_VER to vtexplain/Dockerfile #7467
    • Enable -mysql_server_version in vttestserver, and utilize it in vttestserver container images #7474
    • [vtctld | tests only] testtmclient refactor #7518
    • CI: skip some tests on forked repos #7527
    • Workflow to check make sizegen #7535
    • Add mysqlctl docker image #7557
    • Restore CI workflow shard 26, accidentally dropped #7569
    • Update CODEOWNERS #7586
    • CI: ci-workflow-gen turn string to array to reduce conflicts #7582
    • Add percona-toolkit (for pt-osc/pt-online-schema-change) to the docker/lite images #7603
    • CI: Use ubuntu-18.04 in tests #7614
    • [vttestserver] Fix to work with sharded keyspaces #7617
    • Add tools.go #7517
    • Make vttestserver compatible with persistent data directories #7718
    • Add vtorc binary for rpm,deb builds #7750
    • Fixes bug that prevents creation of logs directory #7761
    • [Java] Guava update to 31.1.1 #7764

    Functionality Neutral Changes

    • VTGate: Remove unused key.Destination.IsUnique() #7565
    • VTGate: Add information_schema query on prepare statement #7746
    • VTGate: Tests for numeric_precision and numeric_scale columns in information_schema #7763
    • Disable flaky test until it can be fixed #7623
    • Tests: reset stat at the beginning of test #7644
    • Cleanup mysql server_test #7645
    • vttablet: fix flaky tests #7543
    • Removed unused tests for Wordpress installation #7516
    • Fix unit test fail after merge #7550
    • Add test with NULL input values for vindexes that did not have any. #7552
    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.0-rc1-29a494f.tar.gz(371.02 MB)
    vitess-10.0.0_rc1-29a494f.x86_64.rpm(368.74 MB)
    vitess_10.0.0-rc1-29a494f_amd64.deb(371.03 MB)
  • v9.0.0(Jan 26, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Incompatible Changes

    The following PRs made changes to behaviors that clients might rely on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.

    • The update to golang 1.15 (#7204) might break systems that use TLS certificates with a common name. A fix is documented here (https://github.com/golang/go/issues/40748#issuecomment-673612108)

    Vitess 9.0 is not compatible with the previous release of the Vitess Kubernetes Operator (2.2.0). A new version of the Operator (2.3.0) is available that is compatible.

    Known Issue(s)

    • VReplication errors when a fixed-length binary column is used as the sharding key #8080

    Bugs Fixed

    VTGate / MySQL compatibility

    • Set Global #6957
    • Set udv allow more expressions #6964
    • Bug which caused the connection to not close in case of error writing an error packet #6977
    • Bug fix #7048 for SelectNone in StreamExecute route engine #7050

    Other

    • Binary PK: fix bug where padding of binary columns was being done incorrectly #6963
    • Pad non-fractional part of DECIMAL type #6967
    • Bug fix regression in /healthz #7090
    • Fix metadata related operation hangs when zk down #7228
    • Fix accidentally-broken legacy vtctl output format #7285
    • Healthcheck: use isIncluded correctly to fix replica/rdonly routing bug #6904

    Functionality Added or Changed

    VTGate / MySQL compatibility

    • VTGate: Allow INSERT with all defaults #6969
    • VTGate: Allow YEAR column type with length specified #6975
    • VTGate: Retry Execute when reserved connection is lost #6983
    • VTGate: Use ephemeral buffer when reading rows #6990
    • VTGate: Allow time_zone in reserved connection #6998
    • VTGate: Improved support for UNION #7007
    • VTGate: Add DDL parser support for FULLTEXT indexes. #7001
    • VTGate: Extend comparability of EvalResult to support hash codes #7016
    • VTGate: Use proto equal method #7017
    • VTGate: Fix COM_STMT_EXECUTE packet decode #7020
    • VTGate: Adds Planning and Parsing Support for Create Index of MySQL 5.7 #7024
    • VTGate: UNION DISTINCT support on vtgate #7029
    • VTGate: Improve database ddl plan #7034
    • VTGate: Support for hex & shard in vindex query #7044
    • VTGate: Use distinct primitive to solve more queries #7047
    • VTGate: Route using vindex for composite IN clause #7049
    • VTGate: Optimise struct field layout #7052
    • VTGate: Refactoring of plan building #7054
    • VTGate: Rewrite joins written with the USING construct #6660
    • VTGate: Add option to GetSchema to only send the row count and data length over the wire #6985
    • VTGate: Adds Planning and Parsing Support for Create Database of MySQL 5.7 #7068
    • VTGate: Make sure to check all GROUP BY columns #7080
    • VTGate: Separate sub-query and derived table into different structs #7081
    • VTGate: Adds Planning and Parsing Support for Alter Database of MySQL 5.7 #7086
    • VTGate: Convert usages of DDL struct to DDLStatement interface #7096
    • VTGate: Adds Planning and Parsing Support for Drop Database of MySQL 5.7 #7098
    • VTGate: Restore SHOW SCHEMAS support; fixes #7100 #7102
    • VTGate: Refactor Code to create a separate struct for CREATE TABLE #7116
    • VTGate: Allows for vttestserver and vtcombo to respond to VtGateExecute. #7121
    • VTGate: Support for lock and unlock tables #7139
    • VTGate: Merge SelectDBA routes when possible #7140
    • VTGate: Adds support for all the rails queries using information_schema #7143
    • VTGate: Add support for unary expression in order by column #7163
    • VTGate: Skip query rewriting for dual table #7164
    • VTGate: Refactor Code to create a separate struct for ALTER VSCHEMA #7173
    • VTGate: Refactor Show plans #7185
    • VTGate: Show privilege support #7194
    • VTGate: Planning and Parsing Support for Drop Table, Drop View and Alter View #7178
    • VTGate: Cache only dml and select plans #7196
    • VTGate: Planning and Parsing Support for Alter Table #7199
    • VTGate: Add FindAllShardsInKeyspace to vtctldserver #7201
    • VTGate: Initial implementation of vtctld service #7128
    • VTGate: improve-log: FAILED_PRECONDITION #7215
    • VTGate: Default to false for system settings to be changed per session at the database connection level #7299
    • VTGate: Route INFORMATION_SCHEMA queries #6932
    • VTGate: Adds Planning and Parsing Support for Create Index of MySQL 5.7 #7024
    • VTGate: Log sql which exceeds max memory rows #7055
    • VTGate: Enable Client Session Tracking feature in mysql protocol #6783
    • VTGate: Show columns from table_name targeted like select queries #6825
    • VTGate: This PR adds logic to simplify subquery expressions that are simple to
    • VTGate: Adding MySQL Check Constraints #6865
    • VTGate: Manage read your own writes system settings #6871
    • VTGate: Allow table_schema comparisons #6887
    • VTGate: Additional options support for SELECT INTO and LOAD DATA #6872
    • VTGate: Fixes vtgate which throws an error in case of empty statements #6947
    • VTGate: [Forward Port] #6940 - Fix error handling in olap mode #6949
    • VTGate: Adds Planning and Parsing Support for Create View of MySQL 5.7 #7060
    • VTGate: fix error: cannot run Select on table "dual" #7118
    • VTGate: Allow system table to be set as default database #7150
    • VTGate: Move auto_increment from reserved to non reserved keyword #7162
    • VTGate: Add only expr of aliasedExpr to weightstring function #7165
    • VTGate: [9.0] don't try to compare varchars in vtgate #7271
    • VTGate: Load Data From S3 #6823
    • VTGate: Unnest simple subqueries #6831
    • VTGate: Adding MySQL Check Constraints #6869
    • VTExplain: Add sequence table support for vtexplain #7186
    • VSchema: Support back-quoted names #7073
    • Healthcheck: healthy list should be recomputed when a tablet is removed #7176
    • Healthcheck: Hellcatlk wants to merge 1 commit into master from master #6953

    Set Statement Support

    Set statement support has been added in Vitess. There are some system variables which are disabled by default and can be enabled using flag -enable_system_settings on VTGate. These system variables are set on the mysql server. Because they change the mysql session, using them leads to the Vitess connection no longer using the connection pool and forcing dedicated connections.

    VReplication

    • VReplication: refactored and enhanced support for JSON columns #6829
    • VReplication: Don't update tx timestamp on heartbeat #6930
    • VReplication E2E Tests: Refactored tests for readability and attempting to fix flakiness #6991
    • VRepl/Tablet Picker: improve observability of selected tablet #6999
    • VReplication: Handle comment statement type in vstreamer #7092
    • VReplication e2e: Fine tuned test to reduce flakiness and added more logging to debug future flakiness #7138
    • VReplication: Make relay log size & rows configurable. #6992
    • VReplication: New workflows cli UX. Allow reads/writes to be switched independently #7071
    • VReplication: DropSources: change table rename logic #7230
    • VReplication: MoveTables: delete routing rules and update vschema on Complete and Abort #7234
    • VReplication: V2 Workflow Start: wait for streams to start and report errors if any while starting a workflow #7248
    • VReplication: Ignore temp tables created by onlineddl #7159
    • VReplication: Set time zone to UTC while streaming rows #6845
    • VReplication: Materialization and character sets: Add test to verify/demo a workaround for charset issues while using string functions in filters #6847
    • VReplication: Tool to diagnose vreplication issues in production #6892
    • VReplication: Allow multiple blacklists for master #6816
    • VStreamer Field Event: add allowed values for set/enum #6981
    • VDiff: lock keyspace while snapshoting, restart target in case of errors #7012
    • VDiff: make enums comparable #6880
    • VDiff: add ability to limit number of rows to compare #6890
    • VDiff/Tablet Picker: fix issue where vdiff sleeps occasionally for tablet picker retry interval #6944
    • [vtctld]: fix error state in Workflow Show #6970
    • [vtctld] Workflow command: minor fixes #7008
    • MoveTables: validate that source tables exist, move all tables #7018
    • SwitchWrites bug: reverse replication workflows can have wrong start positions #7169

    VTTablet

    • VTTablet: fast and reliable state transitions #7011
    • VTTablet: don't shutdown on too many connections #7039
    • VTTablet: debug/env page to change variables in real-time #7189
    • VTTablet: Adds better errors when there are timeouts in resource pools #7002
    • VTTablet: Return to re-using server IDs for binlog connections #6941
    • VTTablet: Correctly initialize the TabletType stats #6989
    • Backup: Use pargzip instead of pgzip for compression. #7037
    • Backup: Add s3 server-side encryption and decryption with customer provided key #7088

    OnlineDDL

    • Online DDL: follow ups in multiple trajectories #6901
    • Online DDL: cancel running migrations executed by another tablet #7006
    • OnlineDDL: Adding ddl_strategy session variable #7042
    • Online DDL: ddl_strategy session variable and vtctl command line argument #7045
    • Online DDL: Removing online ddl query hint from ALTER TABLE #7069
    • Online DDL: vtgate -ddl-strategy flag renamed to -ddl_strategy #7074 Automatically retry migration that was interrupted during master failover Automatically terminate migrations run by a failed tablet
    • Online DDL:request_context/migration_context #7082
    • Online DDL: Support CREATE, DROP statements in ApplySchema and online DDL #7083
    • Online DDL: ddl_type column #7097
    • OnlineDDL: "cancel-all" command to cancel all pending migrations in keyspace #7099
    • OnlineDDL: Support vtctl OnlineDDL <keyspace> show <context> #7145
    • OnlineDDL: Normalizing Online-DDL queries #7153
    • Online DDL: ddl_strategy=direct #7172
    • Online DDL: Executor database pool size increase #7206
    • Online DDL: DROP TABLE translated to RENAME TABLE statement #7221
    • Online DDL: Adding @@session_uuid to vtgate; used as 'context' #7263
    • Online DDL: ignore errors if extracted gh-ost binary is identical to installed binary #6928
    • Online DDL: Table lifecycle: skip time hint for unspecified states #7151
    • Online DDL: Migration uses low priority throttling #6830
    • Online DDL: Fix parsing of online-ddl command line options #6900
    • OnlineDDL bugfix: make sure schema is applied on tablet #6910
    • OnlineDDL: request_context/migration_context #7082
    • OnlineDDL: Fix missed rename in onlineddl_test #7148
    • OnlineDDL: Online DDL endtoend tests to support MacOS #7168

    VTadmin

    • VTadmin: Initial vtadmin-api, clusters, and service discovery #7187
    • VTadmin: The tiniest possible first implementation of vtadmin-web #7218
    • VTadmin: Add cluster protos to discovery and vtsql package constructors #7224
    • VTadmin: Add static file service discovery implementation #7229
    • VTadmin: Query vtadmin-api from vtadmin-web with fetch + react-query #7239
    • VTadmin: Move allow_alias option in MySqlFlag enum to precede the aliased IDs #7166
    • [vtctld]: vtctldclient generator #7238

    Other

    • Fix comment typo #6974
    • Fix all occurrences of fmt.Sprint(x) where x is int #7244
    • Fix incorrect comments #7257
    • Fix comment for IDPool #7212
    • IsInternalOperationTableName: see if a table is used internally by vitess #7104
    • Add timeout for mysqld_shutdown #6849
    • Should receive healthcheck updates from all tablets in cells_to_watch #6852
    • Workflow listall with no workflows was missing newline #6853
    • Allow incomplete SNAPSHOT keyspaces #6863

    Examples / Tutorials

    • Examples/operator: fix tags and add vtorc example #7358
    • local docker: copy examples/common into /vt/common to match MoveTables user guide #7252
    • Update docker-compose examples to take advantage of improvements in Vitess #7009

    Documentation

    • Vitess Slack Guidelines v1.0 #6961
    • Do vschema_customer_sharded.json before create_customer_sharded.sql #7210
    • Added readme for the demo example #7226
    • Adding @shlomi-noach to CODEOWNERS #6855
    • Add Rohit Nayak to maintainers #6903
    • 7.0.3 Release Notes #6902
    • 8_0_0 Release Notes #6958
    • Update maintainers of Vitess #7093
    • Updating Email Address #7095
    • Update morgo changes #7105
    • Move PR template to .github directory #7126
    • Fix trivial typo #7179
    • Add @ajm188 + @doeg to CODEOWNERS for vtctld service files #7202
    • Add @ajm188 + @doeg as vtadmin codeowners #7223:w

    Build Environment Changes

    • Clean up plan building test cases #7057
    • Fix unit test error #6953, #6993
    • Fixing the 5.6 builds of vitess/lite #6960
    • Pin mariadb to use mariadb-server-10.2 #6966
    • Replace vitess:base with vitess:lite images for docker-compose services #7004
    • Fix flakey TestParallelRunnerApprovalFirstRunningSecondRunning test #7014
    • Allow custom image tags in compose #7043
    • Support statsd for vitess #7072
    • Add vtctl to make install-local #7125
    • Updating Java unit tests for JDK9+ compatibility #7144
    • Add Go Version to Bootstrap Image #7182
    • Update Vitess v8.0 images #7174
    • Fix broken package ref in UBI docker build #7183
    • Convert CentOS extra packages installation to yum instead of downloading #7188
    • Make docker_local: fix missing mysql_server package #7213
    • Add unit test case to improve test coverage for go/sqltypes/result.go #7227
    • Update Golang to 1.15 #7204
    • Add linter configuration #7247
    • Modify targets to restore behavior of make install #6842
    • Download zookeeper 3.4.14 from archive site #6865
    • Bump junit from 4.12 to 4.13.1 in /java #6870
    • Fix ListBackups for gcp and az to work with root directory #6873
    • Pulling bootstrap resources from vitess-resources #6875
    • [Java] Bump SNAPSHOT version to 9.0 after Vitess release 8.0 #6907
    • Change dependencies for lite builds #6933
    • Truncate logged query in dbconn.go. #6959
    • [GO] go mod tidy #7137
    • goimport proto files correctly #7264
    • Cherry pick version of #7233 for release-9.0 #7265
    • Update Java version to 9.0 #7369
    • Adding curl as dependency #6965

    Functionality Neutral Changes

    • Healthcheck: add unit test for multi-cell replica configurations #6978
    • Healthcheck: Correct Health Check for Non-Serving Types #6908
    • Adds timeout to checking for tablets. #7106
    • Remove deprecated vtctl commands, flags and vttablet rpcs #7115
    • Fixes comment to mention the existence of reference tables. #7122
    • Updated pull request template to add more clarity #7193
    • Redact password #7198
    • action_repository: no need for http.Request #7124
    • Testing version upgrade/downgrade path from/to 8.0 #7323
    • Use context from Go's standard library #7235
    • Update operator.yaml backup engine description #6832
    • Docker - upgrade to Debian Buster #6833
    • Updating azblob to remove directory after removing backup #6836
    • Fixing some flaky tests #6874
    • Flaky test: attempt to fix TestConnection in go/test/endtoend/messaging #6879
    • Stabilize test #6882
    • Tablet streaming health fix: never silently skip health state changes #6885
    • Add owners to /go/mysql #6886
    • Fixes a bug in Load From statement #6911
    • Query consolidator: fix to ignore leading margin comments #6917
    • Updates to Contacts section as Reporting #7023
    • Create pull_request_template #7027
    • Fixed pull request template path #7062

    Backport

    • Backport: [vtctld] Fix accidentally-broken legacy vtctl output format #7292
    • Backport #7276: Vreplication V2 Workflows: rename Abort to Cancel #7339
    • Backport #7297: VStreamer Events: remove preceding zeroes from decimals in Row Events
    • Backport #7255: VReplication DryRun: Report current dry run results for v2 commands #7345
    • Backport #7275: VReplication: Miscellaneous improvements #7349
    • Backport 7342: Workflow Show: use timeUpdated to calculate vreplication lag #7354
    • Backport 7361: vtctl: Add missing err checks for VReplication v2 #7363
    • Backport 7297: VStreamer Events: remove preceding zeroes from decimals in Row Events #7340
    Source code(tar.gz)
    Source code(zip)
    vitess-9.0.0-daa6085.tar.gz(318.07 MB)
    vitess-9.0.0-daa6085.x86_64.rpm(316.22 MB)
    vitess_9.0.0-daa6085_amd64.deb(318.08 MB)
simply SQL Parser for Go ( powered by vitess and TiDB )

vitess-sqlparser Simply SQL and DDL parser for Go (powered by vitess and TiDB ) this library inspired by https://github.com/xwb1989/sqlparser (origina

BlasTrain Co., Ltd. 289 Jul 30, 2022
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置,参数n为秒数,查询结

null 13 Jun 14, 2022
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

PingCAP 261 Jul 14, 2022
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice.

DoltHub 862 Aug 5, 2022
Interactive terminal user interface and CLI for database connections. MySQL, PostgreSQL. More to come.

?? dbui dbui is the terminal user interface and CLI for database connections. It provides features like, Connect to multiple data sources and instance

Kanan Rahimov 96 Jul 31, 2022
Golang restAPI crud project with mySql database.

Golang RestAPI using gorilla/mux Golang restAPI crud project with mySql database. Test Api with Thunder Client vs code beautiful Extension. and use Be

Md Abu. Raihan 6 Mar 26, 2022
A Go rest API project that is following solid and common principles and is connected to local MySQL database.

This is an intermediate-level go project that running with a project structure optimized RESTful API service in Go. API's of that project is designed based on solid and common principles and connected to the local MySQL database.

Kıvanç Aydoğmuş 21 Jun 6, 2022
CRUD API example is written in Go using net/http package and MySQL database.

GoCrudBook CRUD API example is written in Go using net/http package and MySQL database. Requirements Go MySQL Code Editor Project Structure GoCrudBook

Serhat Karabulut 4 Aug 1, 2022
A proxy is database proxy that de-identifies PII for PostgresDB and MySQL

Surf Surf is a database proxy that is capable of de-identifying PII and anonymizing sentive data fields. Supported databases include Postgres, MySQL,

null 1 Dec 14, 2021
[mirror] the database client and tools for the Go vulnerability database

The Go Vulnerability Database golang.org/x/vulndb This repository is a prototype of the Go Vulnerability Database. Read the Draft Design. Neither the

Go 50 Aug 2, 2022
Database - Example project of database realization using drivers and models

database Golang based database realization Description Example project of databa

Denis 1 Feb 10, 2022
a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

siddontang 3.7k Aug 9, 2022
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.8k Aug 5, 2022
A high-performance MySQL proxy

kingshard 中文主页 Overview kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read

Fei Chen 6.1k Aug 8, 2022
Golang MySql binary log replication listener

Go MySql binary log replication listener Pure Go Implementation of MySQL replication protocol. This allow you to receive event like insert, update, de

Pavel <Ven> Gulbin 186 Jul 27, 2022
MySQL replication topology management and HA

orchestrator [Documentation] orchestrator is a MySQL high availability and replication management tool, runs as a service and provides command line ac

null 4.6k Aug 10, 2022
db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

null 23 Jul 5, 2022
一个使 mysql,pgsql 数据库表自动生成 go struct 的工具

db2go 一个使 mysql、pgsql 数据库表自动生成 go struct 的工具 快速使用 将项目放入到GOPATH/src目录下

易水韩 18 Jun 1, 2022