Vitess is a database clustering system for horizontal scaling of MySQL.

Overview

Maven Central Build Status codebeat badge Go Report Card FOSSA Status CII Best Practices Coverage

Vitess

Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding.

By encapsulating shard-routing logic, Vitess allows application code and database queries to remain agnostic to the distribution of data onto multiple shards. With Vitess, you can even split and merge shards as your needs grow, with an atomic cutover step that takes only a few seconds.

Vitess has been a core component of YouTube's database infrastructure since 2011, and has grown to encompass tens of thousands of MySQL nodes.

For more about Vitess, please visit vitess.io.

Vitess has a growing community. You can view the list of adopters here.

Reporting a Problem, Issue ,or Bug

To report a problem the best way to get attention is to create a GitHub issue.

For topics that are better discussed live, please join the Vitess Slack workspace. You may post any questions on the #general channel or join some of the special-interest channels.

Follow Vitess Blog for low-frequency updates like new features and releases.

Security

Reporting Security Vulnerabilities

To report a security vulnerability, please email vitess-maintainers.

See Security for a full outline of the security process.

Security Audit

A third party security audit was performed by Cure53. You can see the full report here.

License

Unless otherwise noted, the Vitess source files are distributed under the Apache Version 2.0 license found in the LICENSE file.

FOSSA Status

Comments
  • workflow: Add horizontal resharding workflow.

    workflow: Add horizontal resharding workflow.

    This is a beta version implementation, which simply supports creating the horizontal workflow through UI and updating progress of each step.

    In the end-to-end test (test/workflow_horizontal_resharding.py), only happy path is tested. To simplify the test, rather than fetching information from front-end and checking explicitly, I manually interacting with the UI after setting up the environments.

    Unit test will be added later.

    Possible Optimizations (wait for feedback from reviewers, the listed points might be useless):

    • make UI more friendly, s.t. the user input doesn't have to input the shard list
    • supports action for the workflow, which allows the user to stop and restart the workflow
    • adding possible corner case end-to-end test

    This change is Reviewable

    opened by wangyipei01 87
  • Experimental: automated, scheduled, dependency free online DDL via gh-ost/pt-online-schema-change

    Experimental: automated, scheduled, dependency free online DDL via gh-ost/pt-online-schema-change

    This PR (work in progress) introduces zero dependency online schema changes with gh-ost/pt-online-schema-change.

    UPDATE: this comment edited to reflect support for pt-online-schema-change. Originally this PR only supported gh-ost. Mostly whenever you see gh-ost, consider pt-online-schema-change to apply, as well.

    TL;DR

    User will issue:

    alter with 'gh-ost' table example modify id bigint not null;
    
    alter with 'pt-osc' table example modify id bigint not null
    

    or

    $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global \
        ApplySchema -sql "alter with 'gh-ost' table example modify id bigint unsigned not null" commerce
    
    $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global \
        ApplySchema -sql "alter with 'pt-osc' table example modify id bigint unsigned not null" commerce
    

    and vitess will schedule an online schema change operation to run on all relevant shards, then proceed to apply the change via gh-ost on all shards.

    While this PR is WIP, this flow works. More breakdown to follow, indicating what's been done and what's still missing.

    The ALTER TABLE problem

    First, to iterate the problem: schema changes have always been a problem with MySQL; a straight ALTER is a blocking operation; a ONLINE ALTER is only "online" on the master/primary, but is effectively blocking on replicas. Online schema change tools like pt-online-schema-change and gh-ost overcome these limitations by emulating an ALTER on a "ghost" table, which is populated from the original table, then swapped in its space.

    For disclosure, I authored gh-ost's code as part of the database infrastructure team at GitHub.

    Traditionally, online schema changes are considered to be "risky". Trigger based migrations add significant load onto the master server, and their cut-over phase is known to be a dangerous point. gh-ost was created at GitHub to address these concerns, and successfully eliminated concerns for operational risks: with gh-ost the load on the master is low, and well controlled, and the cut-over phase is known to cause no locking issues. gh-ost comes with different risks: it applies data changes programmatically, thus the issue of data integrity is of utmost importance. Another note of concern is data traffic: going out from MySQL into gh-ost and back into MySQL (as opposed to all-in MySQL in pt-online-schema-change).

    This way or the other, running an online schema change is typically a manual operation. A human being will schedule the migration, kick it running, monitor it, possibly cut-over. In a sharded environment, a developer's request to ALTER TABLE explodes to n different migrations, each needs to be scheduled, kicked, monitored & tracked.

    Sharded environments are obviously common for vitess users and so these users feel the pain more than others.

    Schema migration cycle & steps

    Schema management is a process that begins with the user designing a schema change, and ends with the schema being applied in production. This is a breakdown of schema management steps as I know them:

    1. Design code
    2. Publish changes (pull request)
    3. Review
    4. Formalize migration command (the specific ALTER TABLE or pt-online-schema-change or gh-ost command)
    5. Locate: where in production should this migration run?
    6. Schedule
    7. Execute
    8. Audit/monitor
    9. Cut-over/complete
    10. Cleanup
    11. Notify user
    12. Deploy & merge

    What we propose to address

    Vitess's architecture uniquely positions it to be able to automate away much of the process. Specifically:

    • Formalize migration command: turning an ALTER TABLE statement into a gh-ost invocation is super useful if done by vitess, since vitess can not only validate schema/params, but also can provide credentials, identify a throttle-control replica, can instruct gh-ost on how to communicate progress via hooks, etc.
    • Locate: given schema/table, vitess just knows where the table is located. It knows if the schema is sharded. It knows who the shards are, who the shards masters are. It knows where to run gh-ost. Last, vitess can tell us which replicas we can use for throttling.
    • Schedule: vitess is again in a unique position to schedule migrations. The fact someone asks for a migration to run does not mean the migration should start right away. For example, a shard may already be running an earlier migration. Running two migrations at a time is less than ideal, and it's best to wait out the first migration before beginning the second. A scheduling mechanism is both useful to running the migrations in optimal order/sequence, as well as providing feedback to the user ("your migration is on hold because this and that", or "your migration is 2nd in queue to run")
    • Execute: vttablet is the ideal entity to run a migration; can read instructions from topo server and can write progress to topo server. vitess is aware of possible master failovers and can request a re-execute is a migration is so interrupted mid process.
    • Audit/monitor: vtctld API can offer endpoints to track status of a migration (e.g. "in progress on -80, in queue on 80-"). It may offer progress pct and ETA.
    • cut-over/complete: in my experience with gh-ost, the cut-over phase is safe to automate away.
    • cleanup: the old table needs to be dropped; vttablet is in an excellent position to automate that away.

    What this PR does, and what we expect to achieve

    The guideline for this PR is: zero added dependencies; everything must be automatically and implicitly available via a normal vitess installation.

    A breakdown:

    User facing

    This PR enables the user to run an online schema migration (aka online DDL) via:

    • vtgate: the user connects to vitess with their standard MySQL client, and issues a ALTER WITH 'gh-ost' TABLE ... statement. Notice this isn't a valid MySQL syntax -- it's a hint for vitess that we want to run this migration online. vitess still supports synchronous, "normal" ALTER TABLE statements, which IMO should be discouraged.
    • vtctl: the user runs vtctl ApplySchema -sql "alter with _gh-ost' table ...".

    The response, in both cases, is a migration ID, or a job ID, if you will. Consider the following examples.

    via vtgate:

    
    mysql> create table example(id int auto_increment primary key, name tinytext);
    
    mysql> show create table example \G
    
    CREATE TABLE `example` (
      `id` int NOT NULL AUTO_INCREMENT,
      `name` tinytext,
      PRIMARY KEY (`id`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    
    mysql> alter with 'gh-ost' table example modify id bigint not null, add column status int, add key status_dx(status);
    +--------------------------------------+
    | uuid                                 |
    +--------------------------------------+
    | 211febfa-da2d-11ea-b490-f875a4d24e90 |
    +--------------------------------------+
    
    -- <wait...>
    
    mysql> show create table example \G
    
    CREATE TABLE `example` (
      `id` bigint NOT NULL,
      `name` tinytext,
      `status` int DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `status_dx` (`status`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    

    via vtctl:

    $ mysql -e "show create table example\G"
    
    CREATE TABLE `example` (
      `id` bigint NOT NULL,
      `name` tinytext,
      `status` int DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `status_dx` (`status`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    
    
    $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global \
        ApplySchema -sql "alter with 'gh-ost'  table example modify id bigint unsigned not null" commerce
    8ec347e1-da2e-11ea-892d-f875a4d24e90
    
    
    $ mysql -e "show create table example\G"
    
    CREATE TABLE `example` (
      `id` bigint unsigned NOT NULL,
      `name` tinytext,
      `status` int DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `status_dx` (`status`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    

    In both cases, a UUID is returned, which can be used for tracking (WIP) the progress of the migration across shards.

    Parser

    Vitess' parser now accepts ALTER WITH 'gh-ost' TABLE and ALTER WITH 'pt-osc' TABLE syntax. We're still to determine if this is the exact syntax we want to go with.

    Topo

    Whether submitted by vtgate or vtctl, we don't immediately run the migration. As mentioned before, we may wish to postpone the migration. Perhaps the relevant servers are already running a migration.

    Instead, we write the migration request into global topo, e.g.:

    • key: /vitess/global/schema-migration/requests/90c5afd4-da38-11ea-a3ff-f875a4d24e90
    • content:
    {"keyspace":"commerce","table":"example","sql":"alter table example modify id bigint not null","uuid":"90c5afd4-da38-11ea-a3ff-f875a4d24e90","online":true,"time_created":1596701930662801294,"status":"requested"}
    

    Once we create the request in topo, we immediately return the generated UUID/migration ID (90c5afd4-da38-11ea-a3ff-f875a4d24e90 in the above example) to the user.

    vtctld

    vtctld gets a conceptual "upgrade" with this PR. It is no longer a reactive service. vtctld now actively monitors new schema-migration/requests in topo.

    ~~When it sees such a request, it evaluates what are the relevant n shards.~~

    ~~With current implementaiton, it writes n "job" entries, one per shard. e.g.~~

    • /vitess/global/schema-migration/jobs/commerce/-80/ce45b84a-da2d-11ea-b490-f875a4d24e90 and /vitess/global/schema-migration/jobs/commerce/80-/ce45b84a-da2d-11ea-b490-f875a4d24e90 for a keyspace with two shards; or just
    • /vitess/global/schema-migration/jobs/commerce/0/1dd17132-da23-11ea-a3d2-f875a4d24e90 for a keyspace with one shard.

    DONE: WIP: we will investigate use of new VExec to actually distribute the jobs to vttablet.

    what vtctld does now, is, once it sees a migration request, it pushes a VExec request for that migration. If the VExec request succeeds, that means all shards have been notified, and vtctld can stow away the migration request (work is complete as far as vtctld is concerned). If VExec returns with an error, that means at least one shard did not get the request, and vtctld will keep retrying pushing this request.

    vttablet

    This is where most of the action takes place.

    vttablet runs a migration service which continuously probes for, schedules, and executes migrations.

    DONE: ~~With current implementation, tablets which have tablet_type=MASTER continuously probe for new entries. We look to replace this with VExec.~~

    migration requests are pushed via VExec; the request includes the INSERT IGNORE query that persists the migration in _vt.schema_migrations. The tablet no longer reads from, nor writes to, Global Topo.

    A new table is introduced: _vt.schema_migrations, which is how vttablet manages and tracks its own migrations.

    vttablet will only run a single migration at a time.

    vttablet will see if there's an unhandled migration requests. It will queue it.

    vttablet will make a migration ready if there's no running migration and no other migration is marked as ready.

    vttablet will run a ready migration. This is really the interesting part, with lots of goodies:

    • vttablet will evaluate the gh-ost ... command to run. It will obviously populate --alter=... --database=....
    • vttablet creates a temp directory where it generates a script to run gh-ost.
    • vttablet creates a hooks path and auto-generates hook files. The hooks will interact with vttablet
    • vttablet has an API endpoint by which the hooks can communicate gh-ost's status (started/running/success/failure) with vttablet.
    • vttablet provides gh-ost with --hooks-hint which is the migration's UUID.
    • vttablet automatically generates a gh-ost user on the MySQL server, with a random password. The password is never persisted and does not appear on ps. It is written to, and loaded from, an environment variable.
    • vttablet grants the proper privileges on the newly created account
    • vttablet will destroy the account once migration completes.
    • vitess repo includes a gh-ost binary. We require gh-ost from openark/gh-ost as opposed to github/gh-ost because we've had to make some special adjustments to gh-ost s oas to support this flow. I do not have direct ownership to github/gh-ost and cannot enforce those changes upstream, though I have made the contribution requestss upstream.
    • make build automatically appends gh-ost binary, compressed, to vttablet binary, via Ricebox.
    • vttablet, upon startup, auto extracts gh-ost binary into /tmp/vt-gh-ost. Please note that the user does not need to install gh-ost.
    • WIP: vttablet to report back the job as complete/failed. We look to use VExec. TBD.

    Tracking breakdown

    • [x] New OnlineDDL struct, defines a migration request and its status
    • [x] Parser supports ALTER WITH 'gh-ost' TABLE and ALTER WITH 'pt-osc' TABLE syntax
    • [x] builder and analyzer to create an Online DDL plan (write to topo)
    • [x] vtctl to skip "big changes" check when -online_schema_change is given
    • [x] tablet_executor to submit an online DDL request to topo as opposed to running it on tablets
    • [x] vtctld runs a daemon to monitor for, and review migration requests
    • [x] vtctld evaluates which shards are affected
    • [x] _vt.schema_migrations backend table to support migration automation (on each shard))
    • [x] vttablet validates MySQL connection and variables
    • [x] vttablet creates migration command
    • [x] vttablet creates hooks
    • [x] vttablet provides HTTP API for hooks to report their status back
    • [x] vttablet creates gh-ost user with random password
    • [x] vttablet destroys gh-ost user upon completion
    • [x] gh-ost embedded in vttablet binary and auto-extracted by vttablet
    • [x] vttablet runs a dry-run execution
    • [x] vttablet runs a --execute (actual) execution
    • [x] vttablet supports a Cancel request (not used yet) to abort migration
    • [x] vttablet as a state machine to work throught the migration steps
    • [x] counters for gh-ost migration requests, suceessful and failed migrations
    • [x] use of VExec to apply migrations onto tablets
    • [x] use of VExec to control migrations (abort, retry)
    • [ ] consider flow for retries
    • [ ] identify a reparent operation that runs during a migration, probabaly auto-restart the migration
    • [ ] vttablet to heuristically check for available disk space
    • [x] tracking, auditing of all migrations
    • [x] getting gh-ost logs if necessary
    • [x] what's the best way to suggest we want an online migration? Does current ALTER WITH 'gh-ost' TABLE... and ALTER WITH 'pt-osc' TABLE syntax make sense? Other?
    • [ ] For first iteration, migrations and Reshard operations should be mutually exclusive. Can't run both at the same time. Next iterations will remove this constraint.
    • [x] ~~throttle by replica~~
    • [ ] ~~wait for replica to catch up with new credentials before starting the migration~~
    • [x] Use vttablet throttler
    • [x] pt-online-schema-change bundled inside vttablet binary
    • [x] support pt-online-schema-change
    • [ ] ~~define foreign key flags for pt-online-schema-change execution~~ - user can define as runtime flags
    • [x] clenaup online-ddl directory after success
    • [ ] control throttling
    • [x] control termination (panic abort)
    • [x] control termination (panic abort) even after vttablet itself crashes
    • [x] pt-online-schema-change passwords are in cleartext. Can we avoid that?
    • [x] vtctl ApplySchema use same WITH 'gh-ost' and WITH 'pt-osc' query hints as in vtgate.
    • [x] support override of gh-ost and pt-online-schema-change paths
    • [x] cleanup pt-osc triggers after migration failure
    • [x] forcibly remove pt-osc triggers on migration cancellation (overlaps with previous bullet, but has stronger guarantee)
    • [x] cleanup pt-osc triggers from stale/zombie pt-osc migration
    • [x] vtctl OnlineDDL command for simple visibility and manipulation. See https://github.com/vitessio/vitess/pull/6547#issuecomment-681879259
    • [x] end to end tests
    • [x] populate artifacts column, suggesting which tables need to be cleaned up after migration

    Quite likely more entries to be added.

    Further reading, resources, acknowledgements

    We're obviously using gh-ost. I use my own openark/gh-ost since I have no ownership of the original https://github.com/github/gh-ost. gh-ost was/is developed by GitHub 2016-2020.

    pt-online-schema-change is part of the popular Percona Toolkit

    The schema migratoin scheduling and tracking work is based on my previous work at GitHub. The implementation in this PR is new and rewritten, but based on concepts that have matured on my work on skeefree. Consider these resources:

    Also:

    • An early presentation on gh-ost

    Initial incarnation of this PR: https://github.com/planetscale/vitess/pull/67; some useful comments on that PR.

    Call for feedback

    We're looking for community's feedback on the above suggestions/flow. Thank you for taking the time to read and respond!

    Type: Enhancement 
    opened by shlomi-noach 65
  • Adding TLS support to the Java client and JDBC driver

    Adding TLS support to the Java client and JDBC driver

    re: Issue #2209

    TLS connections are already supported by VTGate, gRPC, and all of the non-Java clients. For Java to reach parity, support is needed in the grpc-client, client, and jdbc modules under src/github.com/youtube/vitess/java.

    This PR is a work in progress... visible for early review and vetting, but not yet ready to merge.

    At this point, the VitessJDBCUrl class parses an optional parameter, which indicates that a connection should use SSL. The parameter follows standard MySQL JDBC conventions.

    Pending commits will:

    1. Update GrpcClientFactory, to expose a method for creating a TLS-enabled RpcClient in addition to the current plaintext one.
    2. Update VitessVTGateManager, to use this new method when the JDBC URL calls for SSL.
    3. Write unit test coverage for the above, using a vtgateclienttest mock in a style similar to the src/github.com/youtube/vitess/test/encrypted_transport.py script.
    4. End-to-end testing. Update examples/local/vtgate-up.sh, to use TLS with a self-signed cert when an optional parameter is passed. Also update the examples/client_jdbc.sh test script, to likewise use a TLS connection when an optional parameter is passed.

    Any feedback about this approach is definitely welcomed!


    This change is Reviewable

    opened by steve-perkins 63
  • using nginx for vtadmin web

    using nginx for vtadmin web

    Signed-off-by: Priya Bibra [email protected]

    Description

    Need to move away from using the serve library due PRISMA-2022-0039 (CVSS 7.5) - minimatch 3.0.4 It's underlying dependency serve-handler uses a vulnerable version of minimatch and the dependency itself is not well maintained. Recently learned that minimatch is also a dependency of react-dev-utils which uses recursive-readdir; also not well mainained.

    This PR uses a multistage docker build to first build the static content using node/npm and serve it using nginx in the final image. This way, our dependencies and any related vulnerabilities they may have are not leaked into the deployed image.

    Related Issue

    https://github.com/vitessio/vitess/issues/10676

    Testing Notes

    Image was built using docker build -t vtadmin-test, successfully runs on local host after docker run -p 8080:14201 vtadmin-test:latest

    Type: Enhancement Component: VTAdmin 
    opened by pbibra 31
  • Single shard targeting in V3

    Single shard targeting in V3

    There is no way to target a single shard in the V3 APIs.

    We need to target single shards for two reasons:

    1. We want to shard some of our processing in that we have jobs that process an entire single shard at the time.
    2. There's a very large number of scatter/gather functionality that is currently not supported (e.g. limit on joins, order by and so on). Some we can work around on the query level but some we have to build our own scatter/gather logic.

    Item 2 could eventually be addressed by extending scatter/gather in V3 but item 1 can never be addressed by V3.

    opened by tirsen 31
  • easier build

    easier build

    I worked on this because I finished a brew formula to install Vitess on macOS with one command: brew install vitess. Also, there was some conversation on the Slack channel to improve the build process and make it easier.

    The main change in here is: we will push the vendor libs, instead of getting them during the bootstrap.sh script run. Advantages of this approach:

    • We ensure the version of our libs, not just by the vendor.json file, but they actually will follow the history of this repo.
    • Makes the build process a lot easier. With this, I can build Vitess without needing to run boostrap.sh. This is useful for folks that don't need to change the code but just want to build it.
    • Homebrew build process doesn't allow us to do go get during it, as that is insecure. But, as we vendor our dependencies we don't need to go get during build time anymore.

    Drawbacks:

    • when changing the vendor.json file, we need to run go vendor sync or the bootstrap script again, to upload the new version of the lib.
    • This repository will get bigger in git size. (the local size will still be the same, though)

    I think the advantages outweighs the disadvantages, as the ratio of local builds vs changing vendor.json file is bigger.

    At, least at GitHub, thats what we do, we vendored and upload the lib sources to git. Also, open source projects such as Consul do the same.

    Thoughts? cc @sougou @vmg

    opened by arthurnn 30
  • helm: Allow non-leader Orchestrator instances to accept requests

    helm: Allow non-leader Orchestrator instances to accept requests

    Orchestrator 3.0.7 added a proxy that forwards master only requests, so we don’t have to workaround that by having perpetually unready pods via the /api/leader-check endpoint

    cc @shlomi-noach @enisoc

    opened by derekperkins 30
  • Adds table name to the field proto

    Adds table name to the field proto

    https://dev.mysql.com/doc/refman/5.7/en/c-api-data-structures.html

    The table and table_length fields are available in the C api but are not passed through to clients for use. One of our core java client libraries depends on http://docs.oracle.com/javase/7/docs/api/java/sql/ResultSetMetaData.html#getTableName(int), which in the existing Vitess Driver was hardcoded to return null.

    This PR adds table to the Field proto, populates it in the mysql.Fields() function, and allows access in the Java client. All tests updated to account for the new field, and succeeding.

    I've also tested this in our deployment. However, I've not been able to get the full make test to work in my local dev yet, this was tested with make site_test.

    This is my first foray into both go and vitess, so feedback welcome.

    opened by bbeaudreault 28
  • Preserve case of column names

    Preserve case of column names

    When Vitess receives a query, the tokenizer lowercases all column names so they can be string-compared safely throughout the system. This means all column names returned in the eventual query result are lowercase as well. This causes a problem in PHP when users expect to look up columns by name with the same case they requested in the original query:

    $result = query('select Column from table');
    $row = $result->fetch();
    echo $row['Column']; // array key doesn't exist
    echo $row['column']; // works
    

    In Java (JDBC) they address this by specifying that column name lookups are case-insensitive. We recently fixed our Java client to conform to this (https://github.com/youtube/vitess/pull/1572).

    We could do something similar in PHP by implementing ArrayAccess or extending ArrayObject. Each of those has its pros and cons, but either way, the object we return would not be a true array, so things like array_keys() wouldn't work.

    @tslater @Rastusik Can you weigh in on whether this makes sense from a PHP app writer's perspective?

    @sougou

    Type: Feature Request 
    opened by enisoc 27
  • RFC: Deprecate MariaDB Support in 2022 and Remove in 2023

    RFC: Deprecate MariaDB Support in 2022 and Remove in 2023

    Proposal

    Deprecate official MariaDB support in Vitess 14.0 (~June 2022) and remove it entirely in 16.0 (~Feb 2023). This gives Vitess users approximately 2 years where they can continue to use Vitess with MariaDB and offers adequate time to transition to MySQL 8 (MySQL 5.7 is currently scheduled for EOL in 2023).

    Reasoning

    Note: MySQL and Percona Server for MySQL are considered fully equivalent in Vitess.

    The reasons for proposing this are as follows:

    • Vitess only officially supports MariaDB 10.0-10.3 today
    • MariaDB is a hard fork of MySQL and 10.X is a very different database than MySQL 8.X (which is quickly becoming the default with Vitess)
    • Though there was some outreach from the MariaDB Foundation last year to get Vitess working with more recent versions of MariaDB, that effort seems to have been abandoned
    • MariaDB's GTID implementation — which is NOT compatible with MySQL which uses auto generated UUIDs for the unique host identifier — uses domain_id and server_id together as the "unique" host identifier while every MariaDB instance defaults to 0-1 for its unique host identifier which is in NO way unique. MariaDB's GTID implementation also do not use sets (MySQL uses GTID_EXECUTED and GTID_PURGED sets), but only the last seen sequence value from the given host identifier (so e.g. 0-1:20000 vs 15b57a66-e10d-11eb-a4de-7499a366173e:1-20000 in MySQL) — so you have no way to know if two servers have executed the same full set of GTIDs or not and you cannot easily detect drift and ensure consistency and correctness. When you combine these two things, this is a big problem for Vitess, which relies on fairly complex replication topologies. This makes MariaDB unsafe to use with Vitess VReplication, and you will run into problems (surfaced duplicate GTIDs are the least of your problems, the worse issue is undetected drift and inconsistencies within a shard). This is hardly the only problem with MariaDB usage in Vitess, but I assert that this alone makes MariaDB an unsafe and unsuitable choice.

    And most importantly:

    • Vitess is not well tested on MariaDB today within the project nor the user base
      • There are no known Vitess users with MariaDB in production (please let us know if you do!)

    Given all of this, it seems reasonable to discontinue spending the limited human and capital resources available to the Vitess project — which would otherwise go toward fixing bugs and adding features in Vitess — in trying to maintain official MariaDB support (at least for 10.4+) going forward.

    :warning: While we would not officially support MariaDB as a database within Vitess, we would still want to maintain a clear migration path into Vitess for existing MariaDB installations.

    Feedback

    Please note that we want to do what's best for the Vitess project and its community, so your input is crucial! This is a proposal, it is not an announcement of a decision that's been made. If you have any questions, concerns, or other feedback please let us know!

    Type: RFC 
    opened by mattlord 26
  • grpc build problem

    grpc build problem

    While executing ./bootstrap.sh on OSX I get the following error:

    [HOSTCXX] Compiling src/compiler/csharp_generator.cc
    src/compiler/csharp_generator.cc:47:43: error: no member named 'GetReflectionClassName' in namespace 'google::protobuf::compiler::csharp'
    using google::protobuf::compiler::csharp::GetReflectionClassName;
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
    src/compiler/csharp_generator.cc:237:26: error: use of undeclared identifier 'GetReflectionClassName'
                 "umbrella", GetReflectionClassName(service->file()), "index",
                             ^
    2 errors generated.
    make: *** [/Users/yfinkelstein/work/vitess_workspace/dist/grpc/grpc/objs/opt/src/compiler/csharp_generator.o] Error 1
    ERROR: gRPC build failed
    

    I'm following build setup instructions line by line as described in your "How to build".

    opened by yfinkelstein 26
  • support global routing for reference tables

    support global routing for reference tables

    Addresses #11864.

    Description

    Add support for globally routing queries to reference tables.

    Given:

    • Two unsharded keyspace k1 and sharded keyspace k2.
    • Source table k1.r and reference table k2.r.
    • Unsharded table k1.a
    • Sharded table k2.b.
    • An unqualified, globally routed select SELECT a JOIN r
    • An unqualified, globally routed select SELECT b JOIN r
    • An unqualified, globally routed insert INSERT INTO r

    Currently all three queries will fail with https://github.com/vitessio/vitess/blob/4ff02c0f7af35a520bfc90d32e77fc13810dfdd9/go/vt/vtgate/vindexes/vschema.go#L566

    This PR adds limited support for globally routing these three queries:

    • The first select will join k1.a to k1.r.
    • The second select will join k2.b to k2.r.
    • The insert will be routed to the unsharded source table r.

    Details

    TODO

    Checklist

    • [x] Add support for source to table VSchema definition.
    • [x] Treat reference tables and their source as globally unambiguous.
    • [x] Route unqualified INSERT to reference tables back to the source.
    • [x] Join unqualified table to the keyspace-local reference.
    • [ ] Write tests
    • [ ] Add docs
    Type: Enhancement Component: Query Serving 
    opened by maxenglander 0
  • Parse binlog variable length encoded columns

    Parse binlog variable length encoded columns

    The MySQL binlog format uses variable length encoded integers for the number of columns and the metadata length. Before the Vitess parsing would assume only one byte lengths, breaking anything with 251 or more columns.

    See also https://dev.mysql.com/doc/dev/mysql-server/latest/classbinary__log_1_1Table__map__event.html and the referenced https://dev.mysql.com/doc/dev/mysql-server/latest/classbinary__log_1_1Binary__log__event.html#packed_integer for this format.

    This is the MySQL length encoded integer type, also described at https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_dt_integers.html.

    We already have lenEncIntSize to compute the size of the encoded integer, writeLenEncInt to write a length encoded integer and readLenEncInt to read them. This uses all those utility functions.

    I've also marked this for back porting since this seems like a significant bug, although having 251 or more columns is I guess not that common or we'd have hit this before more often.

    @shlomi-noach I think it would be nice to also have a full end-to-end test for online DDL with a table with many columns, but you probably know how to add that much better than I do.

    Related Issue(s)

    Fixes #11870

    Checklist

    • [x] "Backport to:" labels have been added if this change should be back-ported
    • [x] Tests were added or are not required
    • [ ] Documentation was added or is not required
    Type: Bug Component: VReplication Component: Query Serving Backport to: release-14.0 Component: Online DDL Backport to: release-15.0 
    opened by dbussink 2
  • Bug Report: Binlog parser fails on tables with 251 columns or more

    Bug Report: Binlog parser fails on tables with 251 columns or more

    Overview of the Issue

    Whenever a table has 251 or more columns, it will fail when we use bin logs for that table. This happens in vreplication, online DDL etc. This means that any table with that many columns can't be changed with online DDL.

    It results in an error like:

     "error": "vttablet: rpc error: code = Unknown desc = stream (at source tablet) error @ x-x-x-x-x:1-x: Code: INTERNAL\nunexpected metadata end: got 536 was expecting 353 (data=[x])"
    

    There is already a FIXME annotation in the code that this is a known issue, see:

    https://github.com/vitessio/vitess/blob/22067c7a4e334a7e90850a2119135e6b1ae23279/go/mysql/binlog_event_rbr.go#L74-L76

    MySQL variable length encoded integers are defined in https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_dt_integers.html and we already have the code to deal with these, such as reading them in:

    https://github.com/vitessio/vitess/blob/22067c7a4e334a7e90850a2119135e6b1ae23279/go/mysql/encoding.go#L212

    Reproduction Steps

    Create a table, try to run online DDL so that the number of columns is 251 or more and see it fail.

    Binary Version

    Any supported version (and going back to at least Vitess 3 and likely earlier).
    

    Operating System and Environment details

    n/a
    

    Log Fragments

    No response

    Type: Bug Component: Query Serving 
    opened by dbussink 0
  • Bug Report: `BUG: tablet identifier should represent single table: TableSet{0,1}`

    Bug Report: `BUG: tablet identifier should represent single table: TableSet{0,1}`

    Overview of the Issue

    When executing a subquery that contains more than a single table (e.g. by joining two tables), the planner generates the following warnings:

    W1201 19:52:27.315947    7944 SQL_builder.go:79] error in replacing table expression in semtable: Code: INTERNAL
    BUG: tablet identifier should represent single table: TableSet{0,1}
    

    Note that the planner does not fail - it only outputs the error but then ignores it.

    Reproduction Steps

    Use the vschema defined in go/vt/vtgate/planbuilder/testdata/vschemas/schema.json.

    Run the following query:

    select count(*) from (select 1 from music join user on user.id = music.user_id where music.id = 10) as __subquery_for_count
    

    Binary Version

    Built using commit 22067c7a4e334a7e90850a2119135e6b1ae23279.
    

    Operating System and Environment details

    N/A
    

    Log Fragments

    No response

    Type: Bug Component: Query Serving 
    opened by arthurschreiber 3
  • Question: It takes minimum 40s to create a new empty database using vitess/vttestserver:mysql80 - Could this be improved?

    Question: It takes minimum 40s to create a new empty database using vitess/vttestserver:mysql80 - Could this be improved?

    Question

    Creating a database takes a lot longer when using Vitess than other databases (PostgreSQL, MySQL, SQLite, SQL Server ...) MySQL can do that in 5ms where Vitess, on my computer takes minimum 40s on average.

    This can be problematic when writing tests. In our case since creating a database is fast with the other providers we use that to out advantage to isolate our tests. This means creating and dropping a lot of databases which is normally fast but that can increase a lot the total test time when using Vitess because of the slow database creation.

    I guess there are good reasons for why it's slower and I'm interested to know more about this 👀 It would be great if it could be made faster somehow.

    Reproduction: This is the docker-compose file we use at Prisma:

    version: '3.7'
    services:
      vitess-8:
        image: vitess/vttestserver:[email protected]:1f8c98f716649ac095bae78ead88f522eaf236e9e03baa770b6690d61cbbe29a
        restart: always
        ports:
          - 33807:33807
        healthcheck:
          test: ['CMD', 'mysqladmin', 'ping', '-h127.0.0.1', '-P33577']
          interval: 5s
          timeout: 2s
          retries: 5
          start_period: 25s
        environment:
          PORT: '33804'
          KEYSPACES: 'unsharded'
          NUM_SHARDS: '1'
          MYSQL_BIND_HOST: '0.0.0.0'
          FOREIGN_KEY_MODE: 'disallow'
    

    Which I run with

    docker compose up -d vitess-8
    

    Using TablePlus I connect with the following connection string

    mysql://root:[email protected]:33807
    

    And then execute this minimalistic SQL DDL

    CREATE DATABASE `bonjour-vitess`;
    

    It takes between 40 and 50 seconds on average, which feels very slow in comparison to MySQL which completes in 5ms

    Here are the logs from Docker

    2022-12-01 14:30:45 I1201 13:30:45.176408    1575 syslogger.go:129] bonjour-vitess [keyspace] created value: 
    2022-12-01 14:30:45 I1201 13:30:45.177399    1575 locks.go:245] Locking keyspace bonjour-vitess for action CreateShard
    2022-12-01 14:30:45 I1201 13:30:45.178230    1575 syslogger.go:129] bonjour-vitess/0 [shard] created value: is_primary_serving:true
    2022-12-01 14:30:45 I1201 13:30:45.178319    1575 locks.go:284] Unlocking keyspace bonjour-vitess for successful action CreateShard
    2022-12-01 14:30:45 I1201 13:30:45.184855    1575 tablet_map.go:90] Creating PRIMARY tablet test-0000000016 for bonjour-vitess/0
    2022-12-01 14:30:45 I1201 13:30:45.186385    1575 throttler.go:260] Throttler: initializing config
    2022-12-01 14:30:45 I1201 13:30:45.188465    1575 query_engine.go:204] Stream consolidator is enabled with query size set to 2097152 and total size set to 134217728.
    2022-12-01 14:30:45 I1201 13:30:45.188836    1575 tx_throttler.go:97] Initialized transaction throttler with config: &{enabled:false topoServer:<nil> throttlerConfig:<nil> healthCheckCells:[]}
    2022-12-01 14:30:45 I1201 13:30:45.190245    1575 tm_init.go:483] Reading/creating keyspace and shard records for bonjour-vitess/0
    2022-12-01 14:30:45 I1201 13:30:45.190964    1575 locks.go:245] Locking keyspace bonjour-vitess for action RebuildKeyspace
    2022-12-01 14:30:45 I1201 13:30:45.191747    1575 locks.go:284] Unlocking keyspace bonjour-vitess for successful action RebuildKeyspace
    2022-12-01 14:30:45 I1201 13:30:45.192147    1575 tm_init.go:550] Keyspace rebuilt: bonjour-vitess
    2022-12-01 14:30:45 I1201 13:30:45.192191    1575 vschema_manager.go:102] Received vschema update
    2022-12-01 14:30:45 E1201 13:30:45.192008    1575 tablet.go:244] unable to connect to tablet cell:"test" uid:16: node doesn't exist: tablets/test-0000000016/Tablet
    2022-12-01 14:30:45 I1201 13:30:45.202485    1575 syslogger.go:129] bonjour-vitess/0/test-0000000016 [tablet] created
    2022-12-01 14:30:45 I1201 13:30:45.202625    1575 tablegc.go:167] TableGC: init
    2022-12-01 14:30:45 W1201 13:30:45.202924    1575 tm_init.go:872] cannot start replication during initialization: shard bonjour-vitess/0 has no primary.
    2022-12-01 14:30:45 I1201 13:30:45.202954    1575 tablegc.go:253] TableGC: operating
    2022-12-01 14:30:45 I1201 13:30:45.202990    1575 shard_sync.go:79] Change to tablet state
    2022-12-01 14:30:45 I1201 13:30:45.203920    1575 replmanager.go:87] Replication Manager: starting
    2022-12-01 14:30:45 I1201 13:30:45.228975    1575 replmanager.go:133] Replication is stopped, reconnecting to primary.
    2022-12-01 14:30:45 I1201 13:30:45.229219    1575 replmanager.go:140] Failed to reconnect to primary: no primary tablet for shard bonjour-vitess/0, will keep retrying.
    2022-12-01 14:30:45 I1201 13:30:45.229411    1575 state_manager.go:215] Starting transition to REPLICA Serving, timestamp: 0001-01-01 00:00:00 +0000 UTC
    2022-12-01 14:30:45 I1201 13:30:45.229520    1575 executor.go:337] onlineDDL Executor - Acquiring lock - initMutex
    2022-12-01 14:30:45 I1201 13:30:45.229544    1575 executor.go:339] onlineDDL Executor - Acquired lock - initMutex
    2022-12-01 14:30:45 I1201 13:30:45.229575    1575 tablegc.go:218] TableGC - started execution of Close. Acquiring initMutex lock
    2022-12-01 14:30:45 I1201 13:30:45.229589    1575 tablegc.go:220] TableGC - acquired lock
    2022-12-01 14:30:45 I1201 13:30:45.229612    1575 tablegc.go:223] TableGC - no collector is open
    2022-12-01 14:30:45 I1201 13:30:45.229647    1575 engine.go:90] messager Engine - started execution of Close. Acquiring mu lock
    2022-12-01 14:30:45 I1201 13:30:45.229675    1575 engine.go:92] messager Engine - acquired mu lock
    2022-12-01 14:30:45 I1201 13:30:45.229688    1575 engine.go:95] messager Engine is not open
    2022-12-01 14:30:45 I1201 13:30:45.232230    1575 engine.go:177] Schema Engine: opening
    2022-12-01 14:30:45 I1201 13:30:45.248063    1575 engine.go:155] VStreamer: opening
    2022-12-01 14:30:45 I1201 13:30:45.248113    1575 query_engine.go:271] Query Engine: opening
    2022-12-01 14:30:45 I1201 13:30:45.256106    1575 tx_engine.go:157] TxEngine transition: AcceptingReadOnly
    2022-12-01 14:30:45 I1201 13:30:45.256170    1575 stateful_connection_pool.go:77] Starting transaction id: {1669901445189122003}
    2022-12-01 14:30:45 I1201 13:30:45.256312    1575 repltracker.go:108] Replication Tracker: going into non-primary mode
    2022-12-01 14:30:45 I1201 13:30:45.256413    1575 state_manager.go:588] TabletServer transition: REPLICA: Not connected to mysql -> REPLICA: Serving for tablet :bonjour-vitess/0
    2022-12-01 14:30:45 I1201 13:30:45.256442    1575 state_manager.go:664] Replication is healthy
    2022-12-01 14:30:45 I1201 13:30:45.256478    1575 state_manager.go:688] State: exiting lameduck
    2022-12-01 14:30:45 I1201 13:30:45.256532    1575 tm_state.go:408] Publishing state: alias:{cell:"test" uid:16} port_map:{key:"grpc" value:9016} port_map:{key:"vt" value:8016} keyspace:"bonjour-vitess" shard:"0" type:REPLICA db_name_override:"vt_bonjour-vitess_0" mysql_port:33806
    2022-12-01 14:30:45 I1201 13:30:45.257227    1575 syslogger.go:129] bonjour-vitess/0/test-0000000016 [tablet] updated
    2022-12-01 14:30:45 I1201 13:30:45.257301    1575 tm_state.go:185] Changing Tablet Type: PRIMARY
    2022-12-01 14:30:45 I1201 13:30:45.257441    1575 syslogger.go:129] bonjour-vitess/0/test-0000000016 [tablet] updated
    2022-12-01 14:30:45 I1201 13:30:45.258133    1575 state_manager.go:215] Starting transition to PRIMARY Serving, timestamp: 2022-12-01 13:30:45.257319753 +0000 UTC
    2022-12-01 14:30:45 I1201 13:30:45.259707    1575 engine.go:155] VStreamer: opening
    2022-12-01 14:30:45 I1201 13:30:45.259754    1575 repltracker.go:92] Replication Tracker: going into primary mode
    2022-12-01 14:30:45 I1201 13:30:45.259789    1575 tx_engine.go:157] TxEngine transition: AcceptReadWrite
    2022-12-01 14:30:45 I1201 13:30:45.259818    1575 tx_engine.go:298] TxEngine - called shutdownLocked
    2022-12-01 14:30:45 I1201 13:30:45.259840    1575 tx_engine.go:307] TxEngine - waiting for begin requests
    2022-12-01 14:30:45 I1201 13:30:45.259853    1575 tx_engine.go:309] TxEngine - acquiring state lock again
    2022-12-01 14:30:45 I1201 13:30:45.259867    1575 tx_engine.go:311] TxEngine - state lock acquired again
    2022-12-01 14:30:45 I1201 13:30:45.259882    1575 tx_engine.go:315] TxEngine - stop watchdog
    2022-12-01 14:30:45 I1201 13:30:45.259919    1575 tx_engine.go:361] TxEngine - waiting for empty txPool
    2022-12-01 14:30:45 I1201 13:30:45.259961    1575 tx_engine.go:366] TxEngine - making sure the goroutine has returned
    2022-12-01 14:30:45 I1201 13:30:45.260074    1575 tx_engine.go:331] Immediate shutdown: rolling back now.
    2022-12-01 14:30:45 I1201 13:30:45.260143    1575 tx_engine.go:369] TxEngine - closing the txPool
    2022-12-01 14:30:45 I1201 13:30:45.260188    1575 pool.go:159] connpool - started execution of Close
    2022-12-01 14:30:45 I1201 13:30:45.260213    1575 pool.go:161] connpool - found the pool
    2022-12-01 14:30:45 I1201 13:30:45.260240    1575 pool.go:168] connpool - calling close on the pool
    2022-12-01 14:30:45 I1201 13:30:45.260274    1575 pool.go:170] connpool - acquiring lock
    2022-12-01 14:30:45 I1201 13:30:45.260299    1575 pool.go:172] connpool - acquired lock
    2022-12-01 14:30:45 I1201 13:30:45.260314    1575 pool.go:175] connpool - closing dbaPool
    2022-12-01 14:30:45 I1201 13:30:45.260334    1575 pool.go:177] connpool - finished execution of Close
    2022-12-01 14:30:45 I1201 13:30:45.260345    1575 pool.go:159] connpool - started execution of Close
    2022-12-01 14:30:45 I1201 13:30:45.260355    1575 pool.go:161] connpool - found the pool
    2022-12-01 14:30:45 I1201 13:30:45.260366    1575 pool.go:168] connpool - calling close on the pool
    2022-12-01 14:30:45 I1201 13:30:45.260383    1575 pool.go:170] connpool - acquiring lock
    2022-12-01 14:30:45 I1201 13:30:45.260404    1575 pool.go:172] connpool - acquired lock
    2022-12-01 14:30:45 I1201 13:30:45.260419    1575 pool.go:175] connpool - closing dbaPool
    2022-12-01 14:30:45 I1201 13:30:45.260444    1575 pool.go:177] connpool - finished execution of Close
    2022-12-01 14:30:45 I1201 13:30:45.260465    1575 tx_engine.go:371] TxEngine - closing twoPC
    2022-12-01 14:30:45 I1201 13:30:45.260481    1575 pool.go:159] connpool - started execution of Close
    2022-12-01 14:30:45 I1201 13:30:45.260491    1575 pool.go:161] connpool - found the pool
    2022-12-01 14:30:45 I1201 13:30:45.260505    1575 pool.go:163] connpool - pool is empty
    2022-12-01 14:30:45 I1201 13:30:45.260522    1575 tx_engine.go:373] TxEngine - finished shutdownLocked
    2022-12-01 14:30:45 I1201 13:30:45.260600    1575 stateful_connection_pool.go:77] Starting transaction id: {1669901445189122003}
    2022-12-01 14:30:45 I1201 13:30:45.260770    1575 engine.go:82] Messager: opening
    2022-12-01 14:30:45 I1201 13:30:45.260908    1575 tablegc.go:187] TableGC: opening
    2022-12-01 14:30:45 I1201 13:30:45.263020    1575 state_manager.go:588] TabletServer transition: REPLICA: Serving -> PRIMARY: Serving, Dec 1, 2022 at 13:30:45 (UTC) for tablet :bonjour-vitess/0
    2022-12-01 14:30:45 I1201 13:30:45.263084    1575 state_manager.go:688] State: exiting lameduck
    2022-12-01 14:30:45 I1201 13:30:45.263113    1575 tm_state.go:408] Publishing state: alias:{cell:"test" uid:16} port_map:{key:"grpc" value:9016} port_map:{key:"vt" value:8016} keyspace:"bonjour-vitess" shard:"0" type:PRIMARY db_name_override:"vt_bonjour-vitess_0" mysql_port:33806 primary_term_start_time:{seconds:1669901445 nanoseconds:257319753}
    2022-12-01 14:30:45 I1201 13:30:45.263572    1575 syslogger.go:129] bonjour-vitess/0/test-0000000016 [tablet] updated
    2022-12-01 14:30:45 I1201 13:30:45.263818    1575 shard_sync.go:79] Change to tablet state
    2022-12-01 14:30:45 I1201 13:30:45.263988    1575 shard_sync.go:192] Updating shard record: primary_alias=test-0000000016, primary_term_start_time=2022-12-01 13:30:45.257319753 +0000 UTC
    2022-12-01 14:30:45 I1201 13:30:45.264279    1575 syslogger.go:129] bonjour-vitess/0 [shard] updated value: primary_alias:{cell:"test" uid:16} primary_term_start_time:{seconds:1669901445 nanoseconds:257319753} is_primary_serving:true
    2022-12-01 14:30:45 I1201 13:30:45.264368    1575 shard_watcher.go:36] Starting shard watch of bonjour-vitess/0
    2022-12-01 14:30:45 I1201 13:30:45.267908    1575 tablet_map.go:90] Creating REPLICA tablet test-0000000017 for bonjour-vitess/0
    2022-12-01 14:30:45 I1201 13:30:45.268607    1575 throttler.go:260] Throttler: initializing config
    2022-12-01 14:30:45 I1201 13:30:45.269628    1575 query_engine.go:204] Stream consolidator is enabled with query size set to 2097152 and total size set to 134217728.
    2022-12-01 14:30:45 I1201 13:30:45.269825    1575 tx_throttler.go:97] Initialized transaction throttler with config: &{enabled:false topoServer:<nil> throttlerConfig:<nil> healthCheckCells:[]}
    2022-12-01 14:30:45 I1201 13:30:45.270424    1575 tm_init.go:483] Reading/creating keyspace and shard records for bonjour-vitess/0
    2022-12-01 14:30:45 E1201 13:30:45.270725    1575 tablet.go:244] unable to connect to tablet cell:"test" uid:17: node doesn't exist: tablets/test-0000000017/Tablet
    2022-12-01 14:30:45 I1201 13:30:45.279611    1575 syslogger.go:129] bonjour-vitess/0/test-0000000017 [tablet] created
    2022-12-01 14:30:45 I1201 13:30:45.279696    1575 tablegc.go:167] TableGC: init
    2022-12-01 14:30:45 I1201 13:30:45.279974    1575 shard_sync.go:79] Change to tablet state
    2022-12-01 14:30:45 I1201 13:30:45.280002    1575 tm_init.go:891] Getting a new durability policy for none
    2022-12-01 14:30:45 I1201 13:30:45.280080    1575 tablegc.go:253] TableGC: operating
    2022-12-01 14:30:45 I1201 13:30:45.280741    1575 replmanager.go:87] Replication Manager: starting
    2022-12-01 14:30:45 I1201 13:30:45.304479    1575 replmanager.go:133] Replication is stopped, reconnecting to primary.
    2022-12-01 14:30:45 I1201 13:30:45.304905    1575 locks.go:433] Locking shard bonjour-vitess/0 for action repairReplication to test-0000000016 as parent)
    2022-12-01 14:30:45 I1201 13:30:45.328673    1575 locks.go:476] Unlocking shard bonjour-vitess/0 for successful action repairReplication to test-0000000016 as parent)
    2022-12-01 14:30:45 I1201 13:30:45.328746    1575 replmanager.go:144] Successfully reconnected to primary.
    2022-12-01 14:30:45 I1201 13:30:45.328899    1575 state_manager.go:215] Starting transition to REPLICA Serving, timestamp: 0001-01-01 00:00:00 +0000 UTC
    2022-12-01 14:30:45 I1201 13:30:45.328977    1575 executor.go:337] onlineDDL Executor - Acquiring lock - initMutex
    2022-12-01 14:30:45 I1201 13:30:45.328992    1575 executor.go:339] onlineDDL Executor - Acquired lock - initMutex
    2022-12-01 14:30:45 I1201 13:30:45.329005    1575 tablegc.go:218] TableGC - started execution of Close. Acquiring initMutex lock
    2022-12-01 14:30:45 I1201 13:30:45.329025    1575 tablegc.go:220] TableGC - acquired lock
    2022-12-01 14:30:45 I1201 13:30:45.329038    1575 tablegc.go:223] TableGC - no collector is open
    2022-12-01 14:30:45 I1201 13:30:45.329065    1575 engine.go:90] messager Engine - started execution of Close. Acquiring mu lock
    2022-12-01 14:30:45 I1201 13:30:45.329085    1575 engine.go:92] messager Engine - acquired mu lock
    2022-12-01 14:30:45 I1201 13:30:45.329098    1575 engine.go:95] messager Engine is not open
    2022-12-01 14:30:45 I1201 13:30:45.330490    1575 engine.go:177] Schema Engine: opening
    2022-12-01 14:30:45 I1201 13:30:45.346694    1575 engine.go:155] VStreamer: opening
    2022-12-01 14:30:45 I1201 13:30:45.346764    1575 query_engine.go:271] Query Engine: opening
    2022-12-01 14:30:45 I1201 13:30:45.349006    1575 tx_engine.go:157] TxEngine transition: AcceptingReadOnly
    2022-12-01 14:30:45 I1201 13:30:45.349065    1575 stateful_connection_pool.go:77] Starting transaction id: {1669901445269951711}
    2022-12-01 14:30:45 I1201 13:30:45.349256    1575 repltracker.go:108] Replication Tracker: going into non-primary mode
    2022-12-01 14:30:45 I1201 13:30:45.349445    1575 state_manager.go:588] TabletServer transition: REPLICA: Not connected to mysql -> REPLICA: Serving for tablet :bonjour-vitess/0
    2022-12-01 14:30:45 I1201 13:30:45.349509    1575 state_manager.go:664] Replication is healthy
    2022-12-01 14:30:45 I1201 13:30:45.349580    1575 state_manager.go:688] State: exiting lameduck
    2022-12-01 14:30:45 I1201 13:30:45.349616    1575 tm_state.go:408] Publishing state: alias:{cell:"test" uid:17} port_map:{key:"grpc" value:9017} port_map:{key:"vt" value:8017} keyspace:"bonjour-vitess" shard:"0" type:REPLICA db_name_override:"vt_bonjour-vitess_0" mysql_port:33806
    2022-12-01 14:30:45 I1201 13:30:45.350230    1575 syslogger.go:129] bonjour-vitess/0/test-0000000017 [tablet] updated
    2022-12-01 14:30:45 I1201 13:30:45.354025    1575 tablet_map.go:90] Creating RDONLY tablet test-0000000018 for bonjour-vitess/0
    2022-12-01 14:30:45 I1201 13:30:45.354612    1575 throttler.go:260] Throttler: initializing config
    2022-12-01 14:30:45 I1201 13:30:45.355676    1575 query_engine.go:204] Stream consolidator is enabled with query size set to 2097152 and total size set to 134217728.
    2022-12-01 14:30:45 I1201 13:30:45.355830    1575 tx_throttler.go:97] Initialized transaction throttler with config: &{enabled:false topoServer:<nil> throttlerConfig:<nil> healthCheckCells:[]}
    2022-12-01 14:30:45 I1201 13:30:45.356483    1575 tm_init.go:483] Reading/creating keyspace and shard records for bonjour-vitess/0
    2022-12-01 14:30:45 E1201 13:30:45.356750    1575 tablet.go:244] unable to connect to tablet cell:"test" uid:18: node doesn't exist: tablets/test-0000000018/Tablet
    2022-12-01 14:30:45 I1201 13:30:45.372895    1575 syslogger.go:129] bonjour-vitess/0/test-0000000018 [tablet] created
    2022-12-01 14:30:45 I1201 13:30:45.372983    1575 tablegc.go:167] TableGC: init
    2022-12-01 14:30:45 I1201 13:30:45.373202    1575 tablegc.go:253] TableGC: operating
    2022-12-01 14:30:45 I1201 13:30:45.373236    1575 tm_init.go:891] Getting a new durability policy for none
    2022-12-01 14:30:45 I1201 13:30:45.373310    1575 shard_sync.go:79] Change to tablet state
    2022-12-01 14:30:45 I1201 13:30:45.373833    1575 replmanager.go:87] Replication Manager: starting
    2022-12-01 14:30:45 I1201 13:30:45.397280    1575 replmanager.go:133] Replication is stopped, reconnecting to primary.
    2022-12-01 14:30:45 I1201 13:30:45.397584    1575 locks.go:433] Locking shard bonjour-vitess/0 for action repairReplication to test-0000000016 as parent)
    2022-12-01 14:30:45 I1201 13:30:45.421629    1575 locks.go:476] Unlocking shard bonjour-vitess/0 for successful action repairReplication to test-0000000016 as parent)
    2022-12-01 14:30:45 I1201 13:30:45.421709    1575 replmanager.go:144] Successfully reconnected to primary.
    2022-12-01 14:30:45 I1201 13:30:45.421801    1575 state_manager.go:215] Starting transition to RDONLY Serving, timestamp: 0001-01-01 00:00:00 +0000 UTC
    2022-12-01 14:30:45 I1201 13:30:45.421893    1575 executor.go:337] onlineDDL Executor - Acquiring lock - initMutex
    2022-12-01 14:30:45 I1201 13:30:45.421951    1575 executor.go:339] onlineDDL Executor - Acquired lock - initMutex
    2022-12-01 14:30:45 I1201 13:30:45.421966    1575 tablegc.go:218] TableGC - started execution of Close. Acquiring initMutex lock
    2022-12-01 14:30:45 I1201 13:30:45.421983    1575 tablegc.go:220] TableGC - acquired lock
    2022-12-01 14:30:45 I1201 13:30:45.422023    1575 tablegc.go:223] TableGC - no collector is open
    2022-12-01 14:30:45 I1201 13:30:45.422069    1575 engine.go:90] messager Engine - started execution of Close. Acquiring mu lock
    2022-12-01 14:30:45 I1201 13:30:45.422089    1575 engine.go:92] messager Engine - acquired mu lock
    2022-12-01 14:30:45 I1201 13:30:45.422106    1575 engine.go:95] messager Engine is not open
    2022-12-01 14:30:45 I1201 13:30:45.424091    1575 engine.go:177] Schema Engine: opening
    2022-12-01 14:30:45 I1201 13:30:45.439513    1575 engine.go:155] VStreamer: opening
    2022-12-01 14:30:45 I1201 13:30:45.439561    1575 query_engine.go:271] Query Engine: opening
    2022-12-01 14:30:45 I1201 13:30:45.441594    1575 tx_engine.go:157] TxEngine transition: AcceptingReadOnly
    2022-12-01 14:30:45 I1201 13:30:45.441647    1575 stateful_connection_pool.go:77] Starting transaction id: {1669901445355945044}
    2022-12-01 14:30:45 I1201 13:30:45.441793    1575 repltracker.go:108] Replication Tracker: going into non-primary mode
    2022-12-01 14:30:45 I1201 13:30:45.441885    1575 state_manager.go:588] TabletServer transition: RDONLY: Not connected to mysql -> RDONLY: Serving for tablet :bonjour-vitess/0
    2022-12-01 14:30:45 I1201 13:30:45.441910    1575 state_manager.go:664] Replication is healthy
    2022-12-01 14:30:45 I1201 13:30:45.441937    1575 state_manager.go:688] State: exiting lameduck
    2022-12-01 14:30:45 I1201 13:30:45.441966    1575 tm_state.go:408] Publishing state: alias:{cell:"test" uid:18} port_map:{key:"grpc" value:9018} port_map:{key:"vt" value:8018} keyspace:"bonjour-vitess" shard:"0" type:RDONLY db_name_override:"vt_bonjour-vitess_0" mysql_port:33806
    2022-12-01 14:30:45 I1201 13:30:45.442495    1575 syslogger.go:129] bonjour-vitess/0/test-0000000018 [tablet] updated
    2022-12-01 14:30:45 I1201 13:30:45.446197    1575 tablet_map.go:401] File /vt/schema/bonjour-vitess/vschema.json doesn't exist, skipping vschema for keyspace bonjour-vitess
    2022-12-01 14:30:45 I1201 13:30:45.446265    1575 locks.go:245] Locking keyspace bonjour-vitess for action RebuildKeyspace
    2022-12-01 14:30:45 I1201 13:30:45.446710    1575 locks.go:284] Unlocking keyspace bonjour-vitess for successful action RebuildKeyspace
    2022-12-01 14:30:45 I1201 13:30:45.447231    1575 keyspace_events.go:385] created dedicated watcher for keyspace test/bonjour-vitess
    2022-12-01 14:30:50 I1201 13:30:50.203210    1575 tablegc.go:270] TableGC: transition into primary
    2022-12-01 14:31:14 I1201 13:31:14.969765    1575 healthcheck.go:375] Adding tablet to healthcheck: alias:{cell:"test" uid:17} port_map:{key:"grpc" value:9017} port_map:{key:"vt" value:8017} keyspace:"bonjour-vitess" shard:"0" type:REPLICA db_name_override:"vt_bonjour-vitess_0" mysql_port:33806
    2022-12-01 14:31:14 I1201 13:31:14.971227    1575 tablet_health_check.go:111] HealthCheckUpdate(Serving State): tablet: test-17 () serving false => true for bonjour-vitess/0 (REPLICA) reason: healthCheck update
    2022-12-01 14:31:14 I1201 13:31:14.970851    1575 healthcheck.go:375] Adding tablet to healthcheck: alias:{cell:"test" uid:16} port_map:{key:"grpc" value:9016} port_map:{key:"vt" value:8016} keyspace:"bonjour-vitess" shard:"0" type:PRIMARY db_name_override:"vt_bonjour-vitess_0" mysql_port:33806 primary_term_start_time:{seconds:1669901445 nanoseconds:257319753}
    2022-12-01 14:31:14 I1201 13:31:14.972337    1575 tablet_health_check.go:111] HealthCheckUpdate(Serving State): tablet: test-16 () serving false => true for bonjour-vitess/0 (PRIMARY) reason: healthCheck update
    2022-12-01 14:31:14 I1201 13:31:14.973529    1575 keyspace_events.go:272] keyspace event resolved: bonjour-vitess/bonjour-vitess is now consistent (serving: true)
    2022-12-01 14:31:14 I1201 13:31:14.973853    1575 shard_buffer.go:483] disruption in shard bonjour-vitess/0 resolved (serving: true)
    2022-12-01 14:31:14 I1201 13:31:14.979035    1575 healthcheck.go:375] Adding tablet to healthcheck: alias:{cell:"test" uid:18} port_map:{key:"grpc" value:9018} port_map:{key:"vt" value:8018} keyspace:"bonjour-vitess" shard:"0" type:RDONLY db_name_override:"vt_bonjour-vitess_0" mysql_port:33806
    2022-12-01 14:31:14 I1201 13:31:14.979732    1575 tablet_health_check.go:111] HealthCheckUpdate(Serving State): tablet: test-18 () serving false => true for bonjour-vitess/0 (RDONLY) reason: healthCheck update
    2022-12-01 14:31:15 I1201 13:31:15.538577    1575 healthcheck.go:375] Adding tablet to healthcheck: alias:{cell:"test" uid:16} port_map:{key:"grpc" value:9016} port_map:{key:"vt" value:8016} keyspace:"bonjour-vitess" shard:"0" type:PRIMARY db_name_override:"vt_bonjour-vitess_0" mysql_port:33806 primary_term_start_time:{seconds:1669901445 nanoseconds:257319753}
    2022-12-01 14:31:15 I1201 13:31:15.539609    1575 healthcheck.go:375] Adding tablet to healthcheck: alias:{cell:"test" uid:18} port_map:{key:"grpc" value:9018} port_map:{key:"vt" value:8018} keyspace:"bonjour-vitess" shard:"0" type:RDONLY db_name_override:"vt_bonjour-vitess_0" mysql_port:33806
    2022-12-01 14:31:15 I1201 13:31:15.540142    1575 healthcheck.go:375] Adding tablet to healthcheck: alias:{cell:"test" uid:17} port_map:{key:"grpc" value:9017} port_map:{key:"vt" value:8017} keyspace:"bonjour-vitess" shard:"0" type:REPLICA db_name_override:"vt_bonjour-vitess_0" mysql_port:33806
    2022-12-01 14:31:15 I1201 13:31:15.540974    1575 tablet_health_check.go:111] HealthCheckUpdate(Serving State): tablet: test-16 () serving false => true for bonjour-vitess/0 (PRIMARY) reason: healthCheck update
    2022-12-01 14:31:15 I1201 13:31:15.541655    1575 tablet_health_check.go:111] HealthCheckUpdate(Serving State): tablet: test-17 () serving false => true for bonjour-vitess/0 (REPLICA) reason: healthCheck update
    2022-12-01 14:31:15 I1201 13:31:15.541772    1575 tablet_health_check.go:111] HealthCheckUpdate(Serving State): tablet: test-18 () serving false => true for bonjour-vitess/0 (RDONLY) reason: healthCheck update
    2022-12-01 14:31:15 I1201 13:31:15.981610    1575 tracker.go:100] finished loading schema for keyspace bonjour-vitess. Found 0 columns in total across the tables
    2022-12-01 14:31:15 I1201 13:31:15.981751    1575 vschema_manager.go:164] Received schema update
    2022-12-01 14:31:15 I1201 13:31:15.983488    1575 vschema_manager.go:177] Sent vschema to subscriber
    
    

    Every time things stop for a while after

    2022-12-01 14:30:50 I1201 13:30:50.203210    1575 tablegc.go:270] TableGC: transition into primary
    

    Local environment

    • macbook M1 Max, macOS Monterey

    CI environment

    • Ubuntu 20.04 / 22.04
    Type: Question Component: vttestserver Needs Triage 
    opened by Jolg42 4
Releases(v15.0.1)
  • v15.0.1(Nov 29, 2022)

    Release of Vitess v15.0.1

    Major Changes

    Corrupted results for non-full-group-by queries with JOINs

    An issue in versions <= v14.0.3 and <= v15.0.0 that generated corrupted results for non-full-group-by queries with a JOIN is now fixed. The full issue can be found here, and its fix here.

    VtAdmin web folder is missing while installing Vitess with local method

    When we try to install Vitess locally (https://vitess.io/docs/15.0/get-started/local/#install-vitess) on v15.0, we are getting the following error

    npm ERR! enoent ENOENT: no such file or directory, open '/home/web/vtadmin/package.json'
    

    This issue is fixed in 15.0.1. The full issue can be found here, and its fix here.


    The entire changelog for this release can be found here.

    The release includes 25 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @frouioui, @harshit-gangal, @rsajwani, @vitess-bot[bot]

    Source code(tar.gz)
    Source code(zip)
    vitess-15.0.1-13ee9c8.tar.gz(455.06 MB)
    vitess-15.0.1-13ee9c8.x86_64.rpm(453.41 MB)
    vitess_15.0.1-13ee9c8_amd64.deb(455.07 MB)
  • v14.0.4(Nov 29, 2022)

    Release of Vitess v14.0.4

    Major Changes

    Upgrade to go1.18.7

    Vitess v14.0.4 now runs on go1.18.7. The patch release of Go, go1.18.7, was one of the main reasons for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.18.7 (released 2022-10-04) includes security fixes to the archive/tar, net/http/httputil, and regexp packages, as well as bug fixes to the compiler, the linker, and the go/types package.

    Corrupted results for non-full-group-by queries with JOINs

    An issue in versions <= v14.0.3 and <= v15.0.0 that generated corrupted results for non-full-group-by queries with a JOIN is now fixed. The full issue can be found here, and its fix here.


    The entire changelog for this release can be found here.

    The release includes 24 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @dbussink, @frouioui, @harshit-gangal, @systay, @vitess-bot[bot]

    Source code(tar.gz)
    Source code(zip)
    vitess-14.0.4-7fafc94.tar.gz(488.67 MB)
    vitess-14.0.4-7fafc94.x86_64.rpm(486.91 MB)
    vitess_14.0.4-7fafc94_amd64.deb(488.67 MB)
  • v15.0.0(Oct 25, 2022)

    Release of Vitess v15.0.0

    Summary

    Major Changes

    Breaking Changes

    Flags

    • The deprecated --cpu_profile flag has been removed. Please use the --pprof flag instead.
    • The deprecated --mem-profile-rate flag has been removed. Please use --pprof=mem instead.
    • The deprecated --mutex-profile-fraction flag has been removed. Please use --pprof=mutex instead.
    • The deprecated vtgate/vtexplain/vtcombo flag --planner_version has been removed. Please use --planner-version instead.
    • The deprecated flag --master_connect_retry has been removed. Please use --replication_connect_retry instead.
    • vtctl commands that take shard names and ranges as positional arguments (e.g. vtctl Reshard ks.workflow -80 -40,40-80) need to have their positional arguments separated from their flag arguments by a double-dash separator to avoid the new parsing library from mistaking them as flags (e.g. vtctl Reshard ks.workflow -- -80 -40,40-80).
    • The --cell flag in the vtgate binary no longer has a default value. It is a required argument that has to be specified for the binary to run. Please explicitly specify the flag, if dependent on the flag's default value.
    • The --db-config-*-* VTTablet flags were deprecated in v3.0.0. They have now been deleted as part of this release. You must use --db_dba_* now.

    vttablet Flag Deletions

    The following VTTablet flags were deprecated in 7.0. They have now been deleted

    • --queryserver-config-message-conn-pool-size
    • --queryserver-config-message-conn-pool-prefill-parallelism
    • --client-found-rows-pool-size A different existing flag --queryserver-config-transaction-cap will be used instead
    • --transaction_shutdown_grace_period Use --shutdown_grace_period instead
    • --queryserver-config-max-dml-rows
    • --queryserver-config-allowunsafe-dmls
    • --pool-name-prefix
    • --enable-autocommit Autocommit is always allowed

    Vindex Interface

    All the vindex interface methods are changed by adding context.Context as an input parameter.

    E.g:

    Map(vcursor VCursor, .... ) .... 
    	To
    Map(ctx context.Context, vcursor VCursor, .... ) ....
    

    This only affects users who have added their own custom vindex implementation. They are required to change their implementation with these new interface method expectations.

    Deprecations

    LogStats Table and Keyspace deprecated

    Information about which tables are used was being reported through the Keyspace and Table fields on LogStats. For multi-table queries, this output can be confusing, so we have added TablesUsed, that is a string array, listing all tables and which keyspace they are on. Keyspace and Table fields are deprecated and will be removed in the v16 release of Vitess.

    Orchestrator Integration Deprecation

    Orchestrator integration in vttablet has been deprecated. It will continue to work in this release but is liable to be removed in future releases. Consider using VTOrc instead of Orchestrator as VTOrc goes GA in this release.

    Connection Pool Prefill

    The connection pool with prefilled connections have been removed. The pool now does lazy connection creation.

    InitShardPrimary Deprecation

    The vtcltd command InitShardPrimary has been deprecated. Please use PlannedReparentShard instead.

    Command-line syntax deprecations

    vttablet startup flag deletions

    The following VTTablet flags were deprecated in 7.0. They have now been deleted

    • --queryserver-config-message-conn-pool-size
    • --queryserver-config-message-conn-pool-prefill-parallelism
    • --client-found-rows-pool-size --queryserver-config-transaction-cap will be used instead
    • --transaction_shutdown_grace_period Use --shutdown_grace_period instead
    • --queryserver-config-max-dml-rows
    • --queryserver-config-allowunsafe-dmls
    • --pool-name-prefix
    • --enable-autocommit Autocommit is always allowed

    vttablet startup flag deprecations

    • --enable-query-plan-field-caching has been deprecated. It will be removed in v16.
    • --enable_semi_sync has been deprecated. It will be removed in v16. Instead, set the correct durability policy using SetKeyspaceDurabilityPolicy
    • --queryserver-config-pool-prefill-parallelism, --queryserver-config-stream-pool-prefill-parallelism and --queryserver-config-transaction-prefill-parallelism have all been deprecated. They will be removed in v16.
    • --backup_storage_hook has been deprecated, consider using one of the builtin compression algorithms or --external-compressor and --external-decompressor instead.

    vtbackup flag deprecations

    • --backup_storage_hook has been deprecated, consider using one of the builtin compression algorithms or --external-compressor and --external-decompressor instead.

    VTGate

    vtgate --mysql-server-pool-conn-read-buffers

    --mysql-server-pool-conn-read-buffers enables pooling of buffers used to read from incoming connections, similar to the way pooling happens for write buffers. Defaults to off.

    VDiff v2

    Resume Workflow

    We introduced the ability to resume a VDiff2 workflow:

    $ vtctlclient --server=localhost:15999 VDiff --v2 customer.commerce2customer resume 4c664dc2-eba9-11ec-9ef7-920702940ee0
    VDiff 4c664dc2-eba9-11ec-9ef7-920702940ee0 resumed on target shards, use show to view progress
    
    $ vtctlclient --server=localhost:15999 VDiff --v2 customer.commerce2customer show last
    
    VDiff Summary for customer.commerce2customer (4c664dc2-eba9-11ec-9ef7-920702940ee0)
    State:        completed
    RowsCompared: 196
    HasMismatch:  false
    StartedAt:    2022-06-26 22:44:29
    CompletedAt:  2022-06-26 22:44:31
    
    Use "--format=json" for more detailed output.
    
    $ vtctlclient --server=localhost:15999 VDiff --v2 --format=json customer.commerce2customer show last
    {
    	"Workflow": "commerce2customer",
    	"Keyspace": "customer",
    	"State": "completed",
    	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0",
    	"RowsCompared": 196,
    	"HasMismatch": false,
    	"Shards": "0",
    	"StartedAt": "2022-06-26 22:44:29",
    	"CompletedAt": "2022-06-26 22:44:31"
    }
    

    We leverage this resume capability to automatically restart a VDiff2 workflow that encountered a retryable error.

    We also made a number of other enhancements like progress reporting and features needed to make it a full replacement for VDiff v1. You can see more details in the tracking ticket for the VDiff2 feature complete target: https://github.com/vitessio/vitess/issues/10494

    Now that VDiff v2 is feature complete in 15.0, we hope to make it GA in 16.0.

    Please see the VDiff2 documentation for additional information.

    New command line flags and behavior

    vtctl GetSchema --table-schema-only

    The new flag --table-schema-only skips column introspection. GetSchema only returns general schema analysis, and specifically it includes the CREATE TABLE|VIEW statement in the schema field.

    Support for additional compressors and decompressors during backup & restore

    Backup/Restore now allow you many more options for compression and decompression instead of relying on the default compressor(pargzip). There are some built-in compressors which you can use out-of-the-box. Users will need to evaluate which option works best for their use-case. Here are the flags that control this feature

    • --compression-engine-name
    • --external-compressor
    • --external-decompressor
    • --external-compressor-extension
    • --compression-level

    --compression-engine-name specifies the engine used for compression. It can have one of the following values

    • pargzip (Default)
    • pgzip
    • lz4
    • zstd
    • external

    If you want to use any of the built-in compressors, simply set one of the above values other than external for --compression-engine-name. The value specified in --compression-engine-name is saved in the backup MANIFEST, which is later read by the restore process to decide which engine to use for decompression. Default value for engine is 'pargzip'.

    If you would like to use a custom command or external tool for compression/decompression then you need to provide the full command with arguments to the --external-compressor and --external-decompressor flags. --external-compressor-extension flag also needs to be provided so that compressed files are created with the correct extension. If the external command is not using any of the built-in compression engines (i.e. pgzip, pargzip, lz4 or zstd) then you need to set --compression-engine-name to value 'external'.

    Please note that if you want to keep the current behavior then you don't need to provide any of these flags. You can read more about backup & restore [here] (https://vitess.io/docs/15.0/user-guides/operating-vitess/backup-and-restore/).

    If you decided to switch from an external compressor to one of the built-in supported compressors (i.e. pgzip, pargzip, lz4 or zstd) at any point in the future, you will need to do it in two steps.

    • step #1, set --external-compressor and --external-compressor-extension flag values to empty and change --compression-engine-name to desired value.
    • Step #2, after at least one cycle of backup with new configuration, you can set --external-decompressor flag value to empty.

    The reason you cannot change all the values together is because the restore process will then have no way to find out which external decompressor should be used to process the previous backup. Please make sure you have thought out all possible scenarios for restore before transitioning from one compression engine to another.

    Independent OLAP and OLTP transactional timeouts

    --queryserver-config-olap-transaction-timeout specifies the timeout applied to a transaction created within an OLAP workload. The default value is 30 seconds, but this can be raised, lowered, or set to zero to disable the timeout altogether.

    Until now, while OLAP queries would bypass the query timeout, transactions created within an OLAP session would be rolled back --queryserver-config-transaction-timeout seconds after the transaction was started.

    As of now, OLTP and OLAP transaction timeouts can be configured independently of each other.

    The main use case is to run queries spanning a long period of time which require transactional guarantees such as consistency or atomicity.

    Support for specifying group information in calls to VTGate

    --grpc-use-effective-groups allows non-SSL callers to specify groups information for a caller. Until now, you could only specify the caller-id for the security context used to authorize queries. As of now, you can specify the principal of the caller, and any groups they belong to.

    Online DDL changes

    Concurrent vitess migrations

    All Online DDL migrations using the vitess strategy are now eligible to run concurrently, given --allow-concurrent DDL strategy flag. Until now, only CREATE, DROP and REVERT migrations were eligible, and now ALTER migrations are supported, as well. The terms for ALTER migrations concurrency:

    • DDL strategy must be vitess --allow-concurent ...
    • No two migrations can run concurrently on the same table
    • No two ALTERs will copy table data concurrently
    • A concurrent ALTER migration will not start if another ALTER is running and is not ready_to_complete

    The main use case is to run multiple concurrent migrations, all with --postpone-completion. All table-copy operations will run sequentially, but no migration will actually cut-over, and eventually all migrations will be ready_to_complete, continuously tailing the binary logs and keeping up-to-date. A quick and iterative ALTER VITESS_MIGRATION '...' COMPLETE sequence of commands will cut-over all migrations closely together (though not atomically together).

    vtctl command changes.

    All online DDL show commands can now be run with a few additional parameters

    • --order , order migrations in the output by either ascending or descending order of their id fields.
    • --skip , skip specified number of migrations in the output.
    • --limit , limit results to a specified number of migrations in the output.

    New syntax

    The following is now supported:

    ALTER VITESS_MIGRATION COMPLETE ALL
    

    This works on all pending migrations (queued, ready, running) and internally issues a ALTER VITESS_MIGRATION '<uuid>' COMPLETE for each one. The command is useful for completing multiple concurrent migrations (see above) that are open-ended (--postpone-completion).

    Tablet Throttler

    API changes

    API endpoint /debug/vars now exposes throttler metrics, such as number of hits and errors per app per check type. Example:

    $ curl -s http://127.0.0.1:15100/debug/vars | jq . | grep Throttler
      "ThrottlerAggregatedMysqlSelf": 0.191718,
      "ThrottlerAggregatedMysqlShard": 0.960054,
      "ThrottlerCheckAnyError": 27,
      "ThrottlerCheckAnyMysqlSelfError": 13,
      "ThrottlerCheckAnyMysqlSelfTotal": 38,
      "ThrottlerCheckAnyMysqlShardError": 14,
      "ThrottlerCheckAnyMysqlShardTotal": 42,
      "ThrottlerCheckAnyTotal": 80,
      "ThrottlerCheckMysqlSelfSecondsSinceHealthy": 0,
      "ThrottlerCheckMysqlShardSecondsSinceHealthy": 0,
      "ThrottlerProbesLatency": 355523,
      "ThrottlerProbesTotal": 74,
    

    Mysql Compatibility

    System Settings

    Vitess has had support for system settings from release 7.0 onwards, but this support came with some caveats. As soon as a client session changes a default system setting, a mysql connection gets reserved for it. This can sometimes lead to clients running out of mysql connections. Users were instructed to minimize the use of this feature and to try to set the desired system settings as defaults in the mysql config.

    With this release, Vitess can handle system settings changes in a much better way and clients can use them more freely. Vitess now has the ability to pool changed settings without reserving connections for any particular session.

    This feature can be enabled by setting queryserver-enable-settings-pool flag on the vttablet. It is disabled by default. In future releases, we will make this flag enabled by default.

    Lookup Vindexes

    Lookup vindexes now support a new parameter multi_shard_autocommit. If this is set to true, lookup vindex dml queries will be sent as autocommit to all shards instead of being wrapped in a transaction. This is different from the existing autocommit parameter where the query is sent in its own transaction separate from the ongoing transaction if any i.e. begin -> lookup query execs -> commit/rollback

    Durability Policy

    Cross Cell

    A new durability policy cross_cell is now supported. cross_cell durability policy only allows replica tablets from a different cell than the current primary to send semi-sync ACKs. This ensures that any committed write exists in at least 2 tablets belonging to different cells.

    New EXPLAIN format

    FORMAT=vtexplain

    With this new explain format, you can get an output that is very similar to the command line vtexplain app, but from a running vtgate, through a MySQL query.

    VTOrc

    Old UI Removal and Replacement

    The old UI that VTOrc inherited from Orchestrator has been removed. A replacement UI, more consistent with the other Vitess binaries has been created. In order to use the new UI, --port flag has to be provided.

    Along with the UI, the old APIs have also been deprecated. However, some of them have been ported over to the new UI -

    | Old API | New API | Additional notes | |----------------------------------|----------------------------------|-----------------------------------------------------------------------| | /api/problems | /api/problems | The new API also supports filtering using the keyspace and shard name | | /api/disable-global-recoveries | /api/disable-global-recoveries | Functionally remains the same | | /api/enable-global-recoveries | /api/enable-global-recoveries | Functionally remains the same | | /api/health | /debug/health | Functionally remains the same | | /api/replication-analysis | /api/replication-analysis | Functionally remains the same. Output is now JSON format. |

    Apart from these APIs, we also now have /debug/status, /debug/vars and /debug/liveness available in the new UI.

    Configuration Refactor and New Flags

    Since VTOrc was forked from Orchestrator, it inherited a lot of configurations that don't make sense for the Vitess use-case. All of such configurations have been removed.

    VTOrc ignores the configurations that it doesn't understand. So old configurations can be kept around on upgrading and won't cause any issues. They will just be ignored.

    For all the configurations that are kept, flags have been added for them and the flags are the desired way to pass these configurations going forward. The config file will be deprecated and removed in upcoming releases. The following is a list of all the configurations that are kept and the associated flags added.

    | Configurations Kept | Flags Introduced | |:-------------------------------------:|:-------------------------------------:| | SQLite3DataFile | --sqlite-data-file | | InstancePollSeconds | --instance-poll-time | | SnapshotTopologiesIntervalHours | --snapshot-topology-interval | | ReasonableReplicationLagSeconds | --reasonable-replication-lag | | AuditLogFile | --audit-file-location | | AuditToSyslog | --audit-to-backend | | AuditToBackendDB | --audit-to-syslog | | AuditPurgeDays | --audit-purge-duration | | RecoveryPeriodBlockSeconds | --recovery-period-block-duration | | PreventCrossDataCenterPrimaryFailover | --prevent-cross-cell-failover | | LockShardTimeoutSeconds | --lock-shard-timeout | | WaitReplicasTimeoutSeconds | --wait-replicas-timeout | | TopoInformationRefreshSeconds | --topo-information-refresh-duration | | RecoveryPollSeconds | --recovery-poll-duration |

    Apart from configurations, some flags from VTOrc have also been removed -

    • sibling
    • destination
    • discovery
    • skip-unresolve
    • skip-unresolve-check
    • noop
    • binlog
    • statement
    • grab-election
    • promotion-rule
    • skip-continuous-registration
    • enable-database-update
    • ignore-raft-setup
    • tag

    The ideal way to ensure backward compatibility is to remove the flags listed above while on the previous release. Then upgrade VTOrc. After upgrading, remove the config file and instead pass the flags that are introduced.

    Example Upgrade

    If you are running VTOrc with the flags --ignore-raft-setup --clusters_to_watch="ks/0" --config="path/to/config" and the following configuration

    {
      "Debug": true,
      "ListenAddress": ":6922",
      "MySQLTopologyUser": "orc_client_user",
      "MySQLTopologyPassword": "orc_client_user_password",
      "MySQLReplicaUser": "vt_repl",
      "MySQLReplicaPassword": "",
      "RecoveryPeriodBlockSeconds": 1,
      "InstancePollSeconds": 1,
      "PreventCrossDataCenterPrimaryFailover": true
    }
    

    First drop the flag --ignore-raft-setup while on the previous release. So, you'll be running VTOrc with --clusters_to_watch="ks/0" --config="path/to/config" and the same configuration listed above.

    Now you can upgrade your VTOrc version continuing to use the same flags and configurations, and it will continue to work just the same. If you wish to use the new UI, then you can add the --port flag as well.

    After upgrading, you can drop the configuration entirely and use the new flags like --clusters_to_watch="ks/0" --recovery-period-block-duration=1s --instance-poll-time=1s --prevent-cross-cell-failover

    Default Configuration Files

    The default files that VTOrc searches for configurations in have also changed from "/etc/orchestrator.conf.json", "conf/orchestrator.conf.json", "orchestrator.conf.json" to "/etc/vtorc.conf.json", "conf/vtorc.conf.json", "vtorc.conf.json".

    Flags Restructure

    Flags Diff

    In addition to these major streams of work in release-15.0, we have made tremendous progress on VEP-4, aka The Flag Situation, reorganizing our code so that Vitess binaries and their flags are clearly aligned in help text. An immediate win for usability, this positions us well to move on to a viper implementation which will facilitate additional improvements including standardization of flag syntax and runtime configuration reloads. We are also aligning with industry standards regarding the use of flags, ensuring a seamless experience for users migrating from or integrating with other platforms. Below are the changes for each binary.


    The entire changelog for this release can be found here.

    The release includes 595 commits (excluding merges)

    Thanks to all our contributors: @Abirdcfly, @DeathBorn, @GuptaManan100, @K-Kumar-01, @L3o-pold, @Phanatic, @Weijun-H, @ajm188, @arthurschreiber, @arvind-murty, @brirams, @dbussink, @deepthi, @dependabot[bot], @doeg, @frouioui, @harshit-gangal, @mattlord, @maxenglander, @mgale, @notfelineit, @ofiriluz, @olyazavr, @quinox, @rafer, @renatolabs, @rohit-nayak-ps, @rsajwani, @rvrangel, @saunderst, @shlomi-noach, @systay, @vitess-bot[bot], @vmg, @yoheimuta

    Source code(tar.gz)
    Source code(zip)
    vitess-15.0.0-e8c7e27.tar.gz(453.53 MB)
    vitess-15.0.0-e8c7e27.x86_64.rpm(451.91 MB)
    vitess_15.0.0-e8c7e27_amd64.deb(453.55 MB)
  • v15.0.0-rc1(Oct 5, 2022)

    Release of Vitess v15.0.0-rc1

    Summary

    Known Issues

    Major Changes

    Breaking Changes

    Flags

    • The deprecated --cpu_profile flag has been removed. Please use the --pprof flag instead.
    • The deprecated --mem-profile-rate flag has been removed. Please use --pprof=mem instead.
    • The deprecated --mutex-profile-fraction flag has been removed. Please use --pprof=mutex instead.
    • The deprecated vtgate/vtexplain/vtcombo flag --planner_version has been removed. Please use --planner-version instead.
    • The deprecated flag --master_connect_retry has been removed. Please use --replication_connect_retry instead.
    • vtctl commands that take shard names and ranges as positional arguments (e.g. vtctl Reshard ks.workflow -80 -40,40-80) need to have their positional arguments separated from their flag arguments by a double-dash separator to avoid the new parsing library from mistaking them as flags (e.g. vtctl Reshard ks.workflow -- -80 -40,40-80).

    Vindex Interface

    All the vindex interface methods are changed by adding context.Context as an input parameter.

    E.g:

    Map(vcursor VCursor, .... ) .... 
    	To
    Map(ctx context.Context, vcursor VCursor, .... ) ....
    

    This only affects users who have added their own custom vindex implementation. They are required to change their implementation with these new interface method expectations.

    LogStats Table and Keyspace deprecated

    Information about which tables are used was being reported through the Keyspace and Table fields on LogStats. For multi-table queries, this output can be confusing, so we have added TablesUsed, that is a string array, listing all tables and which keyspace they are on. Keyspace and Table fields are deprecated and will be removed in the v16 release of Vitess.

    Orchestrator Integration Deprecation

    Orchestrator integration in vttablet has been deprecated. It will continue to work in this release but is liable to be removed in future releases. Consider using VTOrc instead of Orchestrator as VTOrc goes GA in this release.

    Connection Pool Prefill

    The connection pool with prefilled connections have been removed. The pool now does lazy connection creation. Following flags are deprecated: queryserver-config-pool-prefill-parallelism, queryserver-config-stream-pool-prefill-parallelism, queryserver-config-transaction-prefill-parallelism and will be removed in future version.

    Command-line syntax deprecations

    vttablet startup flag deletions

    The following VTTablet flags were deprecated in 7.0. They have now been deleted

    • --queryserver-config-message-conn-pool-size
    • --queryserver-config-message-conn-pool-prefill-parallelism
    • --client-found-rows-pool-size --queryserver-config-transaction-cap will be used instead
    • --transaction_shutdown_grace_period Use --shutdown_grace_period instead
    • --queryserver-config-max-dml-rows
    • --queryserver-config-allowunsafe-dmls
    • --pool-name-prefix
    • --enable-autocommit Autocommit is always allowed

    vttablet startup flag deprecations

    • --enable-query-plan-field-caching is now deprecated. It will be removed in v16.
    • --enable_semi_sync is now deprecated. It will be removed in v16. Instead, set the correct durability policy using SetKeyspaceDurabilityPolicy
    • --queryserver-config-pool-prefill-parallelism, --queryserver-config-stream-pool-prefill-parallelism and --queryserver-config-transaction-prefill-parallelism have all been deprecated. They will be removed in v16.

    New command line flags and behavior

    vtgate --mysql-server-pool-conn-read-buffers

    --mysql-server-pool-conn-read-buffers enables pooling of buffers used to read from incoming connections, similar to the way pooling happens for write buffers. Defaults to off.

    VDiff2

    We introduced the ability to resume a VDiff2 workflow:

    $ vtctlclient --server=localhost:15999 VDiff --v2 customer.commerce2customer resume 4c664dc2-eba9-11ec-9ef7-920702940ee0
    VDiff 4c664dc2-eba9-11ec-9ef7-920702940ee0 resumed on target shards, use show to view progress
    
    $ vtctlclient --server=localhost:15999 VDiff --v2 customer.commerce2customer show last
    
    VDiff Summary for customer.commerce2customer (4c664dc2-eba9-11ec-9ef7-920702940ee0)
    State:        completed
    RowsCompared: 196
    HasMismatch:  false
    StartedAt:    2022-06-26 22:44:29
    CompletedAt:  2022-06-26 22:44:31
    
    Use "--format=json" for more detailed output.
    
    $ vtctlclient --server=localhost:15999 VDiff --v2 --format=json customer.commerce2customer show last
    {
    	"Workflow": "commerce2customer",
    	"Keyspace": "customer",
    	"State": "completed",
    	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0",
    	"RowsCompared": 196,
    	"HasMismatch": false,
    	"Shards": "0",
    	"StartedAt": "2022-06-26 22:44:29",
    	"CompletedAt": "2022-06-26 22:44:31"
    }
    

    Please see the VDiff2 documentation for additional information.

    New command line flags and behavior

    vtctl GetSchema --table-schema-only

    The new flag --table-schema-only skips column introspection. GetSchema only returns general schema analysis, and specifically it includes the CREATE TABLE|VIEW statement in the schema field.

    Support for additional compressors and decompressors during backup & restore

    Backup/Restore now allow you many more options for compression and decompression instead of relying on the default compressor(pgzip). There are some built-in compressors which you can use out-of-the-box. Users will need to evaluate which option works best for their use-case. Here are the flags that control this feature

    • --compression-engine-name
    • --external-compressor
    • --external-decompressor
    • --external-compressor-extension
    • --compression-level

    --compression-engine-name specifies the engine used for compression. It can have one of the following values

    • pargzip (Default)
    • pgzip
    • lz4
    • zstd
    • external

    where 'external' is set only when using a custom command or tool other than the ones that are already provided. If you want to use any of the built-in compressors, simply set one of the above values for --compression-engine-name. The value specified in --compression-engine-name is saved in the backup MANIFEST, which is later read by the restore process to decide which engine to use for decompression. Default value for engine is 'pgzip'.

    If you would like to use a custom command or external tool for compression/decompression then you need to provide the full command with arguments to the --external-compressor and --external-decompressor flags. --external-compressor-extension flag also needs to be provided so that compressed files are created with the correct extension. If the external command is not using any of the built-in compression engines (i-e pgzip, pargzip, lz4 or zstd) then you need to set --compression-engine-name to value 'external'.

    Please note that if you want the current production behavior then you don't need to change any of these flags. You can read more about backup & restore [here] (https://vitess.io/docs/15.0/user-guides/operating-vitess/backup-and-restore/).

    If you decided to switch from an external compressor to one of the built-in supported compressors (i-e pgzip, pargzip, lz4 or zstd) at any point in the future, you will need to do it in two steps.

    • step #1, set --external-compressor and --external-compressor-extension flag values to empty and change --compression-engine-name to desired value.
    • Step #2, after at least one cycle of backup with new configuration, you can set --external-decompressor flag value to empty.

    The reason you cannot change all the values together is because the restore process will then have no way to find out which external decompressor should be used to process the previous backup. Please make sure you have thought out all possible scenarios for restore before transitioning from one compression engine to another.

    Independent OLAP and OLTP transactional timeouts

    --queryserver-config-olap-transaction-timeout specifies the timeout applied to a transaction created within an OLAP workload. The default value is 30 seconds, but this can be raised, lowered, or set to zero to disable the timeout altogether.

    Until now, while OLAP queries would bypass the query timeout, transactions created within an OLAP session would be rolled back --queryserver-config-transaction-timeout seconds after the transaction was started.

    As of now, OLTP and OLAP transaction timeouts can be configured independently of each other.

    The main use case is to run queries spanning a long period of time which require transactional guarantees such as consistency or atomicity.

    Support for specifying group information in calls to VTGate

    --grpc-use-effective-groups allows non-SSL callers to specify groups information for a caller. Until now, you could only specify the caller-id for the security context used to authorize queries. As of now, you can specify the principal of the caller, and any groups they belong to.

    Online DDL changes

    Concurrent vitess migrations

    All Online DDL migrations using the vitess strategy are now eligible to run concurrently, given --allow-concurrent DDL strategy flag. Until now, only CREATE, DROP and REVERT migrations were eligible, and now ALTER migrations are supported, as well. The terms for ALTER migrations concurrency:

    • DDL strategy must be vitess --allow-concurent ...
    • No two migrations can run concurrently on the same table
    • No two ALTERs will copy table data concurrently
    • A concurrent ALTER migration will not start if another ALTER is running and is not ready_to_complete

    The main use case is to run multiple concurrent migrations, all with --postpone-completion. All table-copy operations will run sequentially, but no migration will actually cut-over, and eventually all migrations will be ready_to_complete, continuously tailing the binary logs and keeping up-to-date. A quick and iterative ALTER VITESS_MIGRATION '...' COMPLETE sequence of commands will cut-over all migrations closely together (though not atomically together).

    vtctl command changes.

    All online DDL show commands can now be run with a few additional parameters

    • --order , order migrations in the output by either ascending or descending order of their id fields.
    • --skip , skip specified number of migrations in the output.
    • --limit , limit results to a specified number of migrations in the output.

    New syntax

    The following is now supported:

    ALTER VITESS_MIGRATION COMPLETE ALL
    

    This works on all pending migrations (queued, ready, running) and internally issues a ALTER VITESS_MIGRATION '<uuid>' COMPLETE for each one. The command is useful for completing multiple concurrent migrations (see above) that are open-ended (--postpone-completion).

    Tablet throttler

    API changes

    API endpoint /debug/vars now exposes throttler metrics, such as number of hits and errors per app per check type. Example:

    $ curl -s http://127.0.0.1:15100/debug/vars | jq . | grep Throttler
      "ThrottlerAggregatedMysqlSelf": 0.191718,
      "ThrottlerAggregatedMysqlShard": 0.960054,
      "ThrottlerCheckAnyError": 27,
      "ThrottlerCheckAnyMysqlSelfError": 13,
      "ThrottlerCheckAnyMysqlSelfTotal": 38,
      "ThrottlerCheckAnyMysqlShardError": 14,
      "ThrottlerCheckAnyMysqlShardTotal": 42,
      "ThrottlerCheckAnyTotal": 80,
      "ThrottlerCheckMysqlSelfSecondsSinceHealthy": 0,
      "ThrottlerCheckMysqlShardSecondsSinceHealthy": 0,
      "ThrottlerProbesLatency": 355523,
      "ThrottlerProbesTotal": 74,
    

    Mysql Compatibility

    System Settings

    Vitess supported system settings from release 7.0 onwards, but it was always with a pinch of salt. As soon as a client session changes a default system setting, the mysql connection gets blocked for it. This leads to clients running out of mysql connections. The clients were instructed to use this to a minimum and try to set those changed system settings as default on the mysql.

    With this release, Vitess can handle system settings changes in a much better way and the clients can use it more freely. Vitess now pools those changed settings and does not reserve it for any particular session.

    This feature can be enabled by setting queryserver-enable-settings-pool flag on the vttablet. It is disabled by default. In future releases, we will make this flag enabled by default.

    Lookup Vindexes

    Lookup vindexes now support a new parameter multi_shard_autocommit. If this is set to true, lookup vindex dml queries will be sent as autocommit to all shards instead of being wrapped in a transaction. This is different from the existing autocommit parameter where the query is sent in its own transaction separate from the ongoing transaction if any i.e. begin -> lookup query execs -> commit/rollback

    Durability Policy

    Cross Cell

    A new durability policy cross_cell is now supported. cross_cell durability policy only allows replica tablets from a different cell than the current primary to send semi-sync ACKs. This ensures that any committed write exists in at least 2 tablets belonging to different cells.

    New EXPLAIN format

    FORMAT=vtexplain

    With this new explain format, you can get an output that is very similar to the command line vtexplain app, but from a running vtgate, through a MySQL query.

    VTOrc

    Old UI Removal and Replacement

    The old UI that VTOrc inherited from Orchestrator has been removed. A replacement UI, more consistent with the other Vitess binaries has been created. In order to use the new UI, --port flag has to be provided.

    Along with the UI, the old APIs have also been deprecated. However, some of them have been ported over to the new UI -

    | Old API | New API | Additional notes | |----------------------------------|----------------------------------|-----------------------------------------------------------------------| | /api/problems | /api/problems | The new API also supports filtering using the keyspace and shard name | | /api/disable-global-recoveries | /api/disable-global-recoveries | Functionally remains the same | | /api/enable-global-recoveries | /api/enable-global-recoveries | Functionally remains the same | | /api/health | /debug/health | Functionally remains the same | | /api/replication-analysis | /api/replication-analysis | Functionally remains the same. Output is now JSON format. |

    Apart from these APIs, we also now have /debug/status, /debug/vars and /debug/liveness available in the new UI.

    Configuration Refactor and New Flags

    Since VTOrc was forked from Orchestrator, it inherited a lot of configurations that don't make sense for the Vitess use-case. All of such configurations have been removed.

    VTOrc ignores the configurations that it doesn't understand. So old configurations can be kept around on upgrading and won't cause any issues. They will just be ignored.

    For all the configurations that are kept, flags have been added for them and the flags are the desired way to pass these configurations going forward. The config file will be deprecated and removed in upcoming releases. The following is a list of all the configurations that are kept and the associated flags added.

    | Configurations Kept | Flags Introduced | |:-------------------------------------:|:-------------------------------------:| | SQLite3DataFile | --sqlite-data-file | | InstancePollSeconds | --instance-poll-time | | SnapshotTopologiesIntervalHours | --snapshot-topology-interval | | ReasonableReplicationLagSeconds | --reasonable-replication-lag | | AuditLogFile | --audit-file-location | | AuditToSyslog | --audit-to-backend | | AuditToBackendDB | --audit-to-syslog | | AuditPurgeDays | --audit-purge-duration | | RecoveryPeriodBlockSeconds | --recovery-period-block-duration | | PreventCrossDataCenterPrimaryFailover | --prevent-cross-cell-failover | | LockShardTimeoutSeconds | --lock-shard-timeout | | WaitReplicasTimeoutSeconds | --wait-replicas-timeout | | TopoInformationRefreshSeconds | --topo-information-refresh-duration | | RecoveryPollSeconds | --recovery-poll-duration |

    Apart from configurations, some flags from VTOrc have also been removed -

    • sibling
    • destination
    • discovery
    • skip-unresolve
    • skip-unresolve-check
    • noop
    • binlog
    • statement
    • grab-election
    • promotion-rule
    • skip-continuous-registration
    • enable-database-update
    • ignore-raft-setup
    • tag

    The ideal way to ensure backward compatibility is to remove the flags listed above while on the previous release. Then upgrade VTOrc. After upgrading, remove the config file and instead pass the flags that are introduced.

    Example Upgrade

    If you are running VTOrc with the flags --ignore-raft-setup --clusters_to_watch="ks/0" --config="path/to/config" and the following configuration

    {
      "Debug": true,
      "ListenAddress": ":6922",
      "MySQLTopologyUser": "orc_client_user",
      "MySQLTopologyPassword": "orc_client_user_password",
      "MySQLReplicaUser": "vt_repl",
      "MySQLReplicaPassword": "",
      "RecoveryPeriodBlockSeconds": 1,
      "InstancePollSeconds": 1,
      "PreventCrossDataCenterPrimaryFailover": true
    }
    

    First drop the flag --ignore-raft-setup while on the previous release. So, you'll be running VTOrc with --clusters_to_watch="ks/0" --config="path/to/config" and the same configuration listed above.

    Now you can upgrade your VTOrc version continuing to use the same flags and configurations, and it will continue to work just the same. If you wish to use the new UI, then you can add the --port flag as well.

    After upgrading, you can drop the configuration entirely and use the new flags like --clusters_to_watch="ks/0" --recovery-period-block-duration=1s --instance-poll-time=1s --prevent-cross-cell-failover

    Default Configuration Files

    The default files that VTOrc searches for configurations in have also changed from "/etc/orchestrator.conf.json", "conf/orchestrator.conf.json", "orchestrator.conf.json" to "/etc/vtorc.conf.json", "conf/vtorc.conf.json", "vtorc.conf.json".


    The entire changelog for this release can be found here.

    The release includes 551 commits (excluding merges)

    Thanks to all our contributors: @Abirdcfly, @DeathBorn, @GuptaManan100, @K-Kumar-01, @L3o-pold, @Phanatic, @Weijun-H, @ajm188, @arthurschreiber, @arvind-murty, @brirams, @dbussink, @deepthi, @dependabot[bot], @doeg, @frouioui, @harshit-gangal, @mattlord, @maxenglander, @mgale, @notfelineit, @ofiriluz, @olyazavr, @quinox, @rafer, @renatolabs, @rohit-nayak-ps, @rsajwani, @rvrangel, @saunderst, @shlomi-noach, @systay, @vitess-bot[bot], @vmg, @yoheimuta

    Source code(tar.gz)
    Source code(zip)
    vitess-15.0.0-rc1-f2e132b.tar.gz(469.05 MB)
    vitess-15.0.0_rc1-f2e132b.x86_64.rpm(467.36 MB)
    vitess_15.0.0-rc1-f2e132b_amd64.deb(469.07 MB)
  • v14.0.3(Sep 30, 2022)

    Release of Vitess v14.0.3

    Major Changes

    Fix VTOrc Discovery

    In patch releases prior to release 14.0.3, if VTOrc is unable to reach the MySQL instance of a vttablet, it was never able to read discover that tablet again. This problem could be resolved by restarting the VTOrc so that it discovers all the tablets again, but in a kubernetes cluster where the pods are eviced frequently, this posed a greater challenge, since some pods when evicted and rescheduled on a different node, would sometimes fail to be discovered by VTOrc. This has problem has been addressed in this patch by the fix https://github.com/vitessio/vitess/pull/10662.


    The entire changelog for this release can be found here.

    The release includes 12 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @frouioui, @harshit-gangal, @mattlord, @vitess-bot[bot]

    Source code(tar.gz)
    Source code(zip)
    vitess-14.0.3-37c684d.tar.gz(489.04 MB)
    vitess-14.0.3-37c684d.x86_64.rpm(487.26 MB)
    vitess_14.0.3-37c684d_amd64.deb(489.04 MB)
  • v14.0.2(Sep 1, 2022)

    Release of Vitess v14.0.2

    Major Changes

    Upgrade to go1.18.5

    Vitess v14.0.2 now runs on go1.18.5. The patch release of Go, go1.18.5, was one of the main reasons for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.18.4 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the linker, the runtime, and the runtime/metrics package.

    End of life of MariadDB 10.2

    Since the end-of-life of MariaDB 10.2, its Docker image is unavailable, and we decided to remove the unit tests using this version of MariaDB. The Pull Request doing this change is available here. You can find more information on the list of supported databases on our documentation website, here.


    The entire changelog for this release can be found here.

    The release includes 23 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @dbussink, @frouioui, @mattlord, @shlomi-noach, @vitess-bot[bot]

    Source code(tar.gz)
    Source code(zip)
    vitess-14.0.2-4d94e3f.tar.gz(489.04 MB)
    vitess-14.0.2-4d94e3f.x86_64.rpm(487.26 MB)
    vitess_14.0.2-4d94e3f_amd64.deb(489.05 MB)
  • v13.0.3(Sep 1, 2022)

    Release of Vitess v13.0.3

    Major Changes

    Upgrade to go1.17.13

    Vitess v13.0.3 now runs on go1.17.13. The patch release of Go, go1.17.13, was one of the main reasons for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.17.13 (released 2022-08-01) includes security fixes to the encoding/gob and math/big packages, as well as bug fixes to the compiler and the runtime.

    End of life of MariadDB 10.2

    Since the end-of-life of MariaDB 10.2, its Docker image is unavailable, and we decided to remove the unit tests using this version of MariaDB. The Pull Request doing this change is available here. This change is documented on our website here.


    The entire changelog for this release can be found here.

    The release includes 17 commits(excluding merges)

    Thanks to all our contributors: @GuptaManan100, @dbussink, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @vitess-bot[bot]

    Source code(tar.gz)
    Source code(zip)
    vitess-13.0.3-e7a04c1.tar.gz(463.55 MB)
    vitess-13.0.3-e7a04c1.x86_64.rpm(461.91 MB)
    vitess_13.0.3-e7a04c1_amd64.deb(463.57 MB)
  • v12.0.6(Sep 1, 2022)

    Release of Vitess v12.0.6

    Major Changes

    Upgrade to go1.17.13

    Vitess v12.0.6 now runs on go1.17.13. The patch release of Go, go1.17.13, was the main reason for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.17.13 (released 2022-08-01) includes security fixes to the encoding/gob and math/big packages, as well as bug fixes to the compiler and the runtime.

    End of life of MariadDB 10.2

    Since the end-of-life of MariaDB 10.2, its Docker image is unavailable, and we decided to remove the unit tests using this version of MariaDB. The Pull Request doing this change is available here. This change is documented on our website here.


    The entire changelog for this release can be found here.

    The release includes 11 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @dbussink, @deepthi, @frouioui, @harshit-gangal, @rohit-nayak-ps

    Source code(tar.gz)
    Source code(zip)
  • v12.0.5(Jul 27, 2022)

    Release of Vitess v12.0.5

    Major Changes

    Upgrade to go1.17.12

    Vitess v12.0.5 now runs on go1.17.12. The patch release of Go, go1.17.12, is the reason for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.17.12 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the runtime, and the runtime/metrics package. See the Go 1.17.12 milestone on our issue tracker for details.


    The entire changelog for this release can be found here.

    The release includes 7 commits (excluding merges)

    Thanks to all our contributors: @deepthi, @frouioui

    Source code(tar.gz)
    Source code(zip)
  • v14.0.1(Jul 26, 2022)

    Release of Vitess v14.0.1

    Known Issues

    Major Changes

    Upgrade to go1.18.4

    Vitess v14.0.1 now runs on go1.18.4. The patch release of Go, go1.18.4, was one of main motivations for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.18.4 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the linker, the runtime, and the runtime/metrics package. See the Go 1.18.4 milestone on our issue tracker for details.


    The entire changelog for this release can be found here.

    The release includes 25 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @deepthi, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @shlomi-noach, @vitess-bot[bot]

    Source code(tar.gz)
    Source code(zip)
    vitess-14.0.1-4c118cb.tar.gz(488.83 MB)
    vitess-14.0.1-4c118cb.x86_64.rpm(487.08 MB)
    vitess_14.0.1-4c118cb_amd64.deb(488.85 MB)
  • v13.0.2(Jul 26, 2022)

    Release of Vitess v13.0.2

    Major Changes

    Upgrade to go1.17.12

    Vitess v13.0.2 now runs on go1.17.12. The patch release of Go, go1.17.12, was one of main motivations for this release as it includes important security fixes to packages used by Vitess. Below is a summary of this patch release. You can learn more here.

    go1.17.12 (released 2022-07-12) includes security fixes to the compress/gzip, encoding/gob, encoding/xml, go/parser, io/fs, net/http, and path/filepath packages, as well as bug fixes to the compiler, the go command, the runtime, and the runtime/metrics package. See the Go 1.17.12 milestone on our issue tracker for details.


    The entire changelog for this release can be found here.

    The release includes 28 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @aquarapid, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @systay, @vitess-bot[bot], @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-13.0.2-60dc8b9.tar.gz(463.35 MB)
    vitess-13.0.2-60dc8b9.x86_64.rpm(461.69 MB)
    vitess_13.0.2-60dc8b9_amd64.deb(463.36 MB)
  • v14.0.0(Jun 28, 2022)

    Release of Vitess v14.0.0

    Summary

    Known Issues

    Major Changes

    Gen4 is now the default planner

    The new planner has been in the works since end of 2020, and it's finally grown enough to be able to become the default planner for Vitess. This means that many more queries are supported on sharded keyspaces, and old queries might get planned better than before. You can always roll back to the earlier planner, either by providing the flag --planner-version=V3 to vtgate, or by adding a comment to individual queries, like so:

    select /*vt+ PLANNER=V3 */ name, count(*) from users
    

    Schema Tracking is no longer experimental

    Schema tracking is now GA. It is enabled by default. Schema tracking allows the Gen4 planner to plan many more queries. This can be turned off if necessary by changing the relevant flags on VTGate and VTTablet.

    New query support

    Support for aggregation across shards

    Vitess can now plan and execute most aggregation queries across multiple shards and/or keyspaces.

    INSERT from SELECT

    Support has been added for inserting new data from SELECT queries. Now you can insert data from a query into a table using a query like:

    insert into tbl (col) select id from users 
    

    UPDATE from SELECT

    Similarly, we have added support for UPDATE with scalar sub-queries. This allows for queries where the updated value is fetched using a subquery, such as this example:

    update tbl set foo = (select count(*) from otherTbl)
    

    Command-line syntax deprecations

    Vitess has begun a transition to a new library for CLI flag parsing. In order to facilitate a smooth transition, certain syntaxes that will not be supported in the future now issue deprecation warnings when used.

    The messages you will likely see, along with explanations and migrations, are:

    "Use of single-dash long flags is deprecated"

    Single-dash usage will be only possible for short flags (e.g. -v is okay, but -verbose is not).

    To migrate, update your CLI scripts from:

    $ vttablet -tablet_alias zone1-100 -init_keyspace mykeyspace ... # old way
    

    To:

    $ vttablet --tablet_alias zone1-100 --init_keyspace mykeyspace ... # new way
    

    "Detected a dashed argument after a position argument."

    As the full deprecation text goes on to (attempt to) explain, mixing flags and positional arguments will change in a future version that will break scripts.

    Currently, when invoking a binary like:

    $ vtctl --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
    

    Everything after the AddCellInfo is treated by package flag as a positional argument, and we then use a sub FlagSet to parse flags specific to the subcommand. So, at the top-level, flag.Args() returns ["AddCellInfo", "--root", "/vitess/global"].

    The library we are transitioning to is more flexible, allowing flags and positional arguments to be interwoven on the command-line. For the above example, this means that we would attempt to parse --root as a top-level flag for the VTCtl binary. This will cause the program to exit on error, because that flag is only defined on the AddCellInfo subcommand.

    In order to transition, a standalone double-dash (literally, --) will cause the new flag library to treat everything following that as a positional argument, and also works with the current flag parsing code we use.

    So, to transition the above example without breakage, update the command to:

    $ vtctl --topo_implementation etcd2 AddCellInfo -- --root "/vitess/global"
    $ # the following will also work
    $ vtctl --topo_implementation etcd2 -- AddCellInfo --root "/vitess/global"
    $ # the following will NOT work, because --topo_implementation is a top-level flag, not a sub-command flag
    $ vtctl -- --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
    

    New command line flags and behavior

    vttablet --heartbeat_on_demand_duration

    --heartbeat_on_demand_duration joins the already existing heartbeat flags --heartbeat_enable and --heartbeat_interval and adds new behavior to heartbeat writes.

    --heartbeat_on_demand_duration takes a duration value, such as 5s.

    The default value for --heartbeat_on_demand_duration is zero, which means the flag is not set and there is no change in behavior.

    When --heartbeat_on_demand_duration has a positive value, then heartbeats are only injected on demand, based on internal requests. For example, when --heartbeat_on_demand_duration=5s, the tablet starts without injecting heartbeats. An internal module, like the lag throttler, may request the heartbeat writer for heartbeats. Starting at that point in time, and for the duration (a lease) of 5s in our example, the tablet will write heartbeats. If no other requests come in during that time, the tablet then ceases to write heartbeats. If more requests for heartbeats come in, the tablet extends the lease for the next 5s following each request. It stops writing heartbeats 5s after the last request is received.

    The heartbeats are generated according to --heartbeat_interval.

    Deprecation of --online_ddl_check_interval

    The flag --online_ddl_check_interval is deprecated and will be removed in v15. It has been unused in v13.

    Removal of --gateway_implementation

    In previous releases, the discoverygateway was deprecated. In Vitess 14 it is now entirely removed, along with the VTGate flag that allowed us to choose a gateway.

    Deprecation of --planner_version

    The flag --planner_version is deprecated and will be removed in v15. Some binaries used --planner_version, and some used --planner-version. This has been made consistent - all binaries that allow you to configure the planner now take --planner-version. All uses of the underscore form have been deprecated and will be removed in v15.

    Online DDL changes

    Online DDL is generally available

    Online DDL is no longer experimental (with the exception of pt-osc strategy). Specifically:

    • Managed schema changes, the scheduler, the backing tables
    • Supporting SQL syntax
    • vitess strategy (online DDL via VReplication)
    • gh-ost strategy (online DDL via 3rd party gh-ost)
    • Recoverable migrations
    • Revertible migrations
    • Declarative migrations
    • Postponed migrations
    • And all other functionality

    Are all considered production-ready.

    pt-osc strategy (online DDL via 3rd party pt-online-schema-change) remains experimental.

    ddl_strategy: 'vitess'

    ddl_strategy now takes the value of vitess to indicate VReplication-based migrations. It is a synonym to online and uses the exact same functionality. The online term will be phased out in the future and vitess will remain the term of preference.

    Example:

    vtctlclient ApplySchema -skip_preflight -ddl_strategy='vitess' -sql "alter table my_table add column my_val int not null default 0" commerce
    

    --singleton-context and REVERT migrations

    It is now possible to submit a migration with --singleton-context strategy flag, while there's a pending (queued or running) REVERT migration that does not have a --singleton-context flag.

    Support for CHECK constraints

    Online DDL operations are more aware of CHECK constraints, and properly handle the limitation where a CHECK's name has to be unique in the schema. As opposed to letting MySQL choose arbitrary names for shadow table's CHECK constraints, Online DDL now generates unique yet deterministic names, such that all shards converge onto the same names.

    Online DDL attempts to preserve the original check's name as a suffix to the generated name, where possible (names are limited to 64 characters).

    Behavior changes

    • vtctl ApplySchema --uuid_list='...' now rejects a migration if an existing migration has the same UUID but with different migration_context.

    Table lifecycle

    Views

    Table lifecycle now supports views. It does not purge rows from views, and does not keep views in EVAC state (they are immediately transitioned to DROP state).

    Fast drops

    On Mysql 8.0.23 or later, the states PURGE and EVAC are automatically skipped, thanks to 8.0.23 improvements to DROP TABLE speed of operation.

    Tablet throttler

    API changes

    Added /throttler/throttled-apps endpoint, which reports back all current throttling instructions. Note, this only reports explicit throttling requests (such as ones submitted by /throtler/throttle-app?app=...). It does not list incidental rejections based on throttle thresholds.

    API endpoint /throttler/throttle-app now accepts a ratio query argument, a floating point value in the range [0..1], where:

    • 0 means "do not throttle at all"
    • 1 means "always throttle"
    • Any number in between is allowed. For example, 0.3 means "throttle with 0.3 probability", i.e. for any given request there's a 30% chance that the request is denied. Overall we can expect about 30% of requests to be denied. Example: /throttler/throttle-app?app=vreplication&ratio=0.25.

    See new SQL syntax for controlling/viewing throttling, under New Syntax.

    New Syntax

    Control and view Online DDL throttling

    We introduce the following syntax to:

    • Start/stop throttling for all Online DDL migrations, in general
    • Start/stop throttling for a particular Online DDL migration
    • View throttler state
    ALTER VITESS_MIGRATION '<uuid>' THROTTLE [EXPIRE '<duration>'] [RATIO <ratio>];
    ALTER VITESS_MIGRATION THROTTLE ALL [EXPIRE '<duration>'] [RATIO <ratio>];
    ALTER VITESS_MIGRATION '<uuid>' UNTHROTTLE;
    ALTER VITESS_MIGRATION UNTHROTTLE ALL;
    SHOW VITESS_THROTTLED_APPS;
    

    The default duration is "infinite" (set as 100 years):

    • Allowed units are (s)ec, (m)in, (h)our

    The ratio is in the range [0..1]:

    • 1 means throttle everything - the app will not make any progress
    • 0 means no throttling at all
    • 0.8 means on 8 out of 10 checks the app makes, it gets refused

    The syntax SHOW VITESS_THROTTLED_APPS is a generic call to the throttler, and returns information about all throttled apps, not specific to migrations.

    The output of SHOW VITESS_MIGRATIONS ... now includes user_throttle_ratio.

    This column is updated "once in a while", while a migration is running. Normally this is once a minute, but can be more frequent. The migration reports back the throttling instruction set by the user while it was running. This column does not indicate any lag-based throttling that might take place based on the throttler configuration. It only reports the explicit throttling value set by the user.

    Heartbeat

    The throttler now checks in with the heartbeat writer to request heartbeats, any time it (the throttler) is asked for a check.

    When --heartbeat_on_demand_duration is not set, there is no change in behavior.

    When --heartbeat_on_demand_duration is set to a positive value, then the throttler ensures that the heartbeat writer generates heartbeats for at least the following duration. This also means at the first throttler check, it's possible that heartbeats are idle, and so the first check will fail. As heartbeats start running, followup checks will get a more accurate lag evaluation and will respond accordingly. In a sense, it's a "cold engine" scenario, where the engine takes time to start up, and then runs smoothly.

    VDiff2

    We introduced a new version of VDiff -- currently marked as EXPERIMENTAL -- that executes the VDiff on vttablets rather than in vtctld. While this is experimental we encourage you to try it out and provide feedback! This input will be invaluable as we improve the feature on the march toward a production-ready version. You can try it out by adding the --v2 flag to your VDiff command. Here's an example:

    $ vtctlclient --server=localhost:15999 VDiff -- --v2 customer.commerce2customer
    VDiff bf9dfc5f-e5e6-11ec-823d-0aa62e50dd24 scheduled on target shards, use show to view progress
    
    $ vtctlclient --server=localhost:15999 VDiff -- --v2 customer.commerce2customer show last
    
    VDiff Summary for customer.commerce2customer (4c664dc2-eba9-11ec-9ef7-920702940ee0)
    State:        completed
    RowsCompared: 196
    HasMismatch:  false
    StartedAt:    2022-06-26 22:44:29
    CompletedAt:  2022-06-26 22:44:31
    
    Use "--format=json" for more detailed output.
    
    $ vtctlclient --server=localhost:15999 VDiff -- --v2 --format=json customer.commerce2customer show last
    {
    	"Workflow": "commerce2customer",
    	"Keyspace": "customer",
    	"State": "completed",
    	"UUID": "4c664dc2-eba9-11ec-9ef7-920702940ee0",
    	"RowsCompared": 196,
    	"HasMismatch": false,
    	"Shards": "0",
    	"StartedAt": "2022-06-26 22:44:29",
    	"CompletedAt": "2022-06-26 22:44:31"
    }
    

    Even before it's marked as production-ready (feature complete and tested widely in 1+ releases), it should be safe to use and is likely to provide much better results for very large tables.

    For additional details please see the RFC, the README, and the VDiff2 documentation.

    Durability Policy

    Deprecation of durability_policy Flag

    The durability policy for a keyspace is now stored in the keyspace record in the topology server. The durability_policy flag used by VTCtl, VTCtld, and VTWorker binaries has been deprecated and will be removed in a future release.

    New and Augmented Commands

    The VTCtld command CreateKeyspace has been augmented to take in an additional argument --durability-policy which will allow users to set the desired durability policy for a keyspace at creation time.

    For existing keyspaces, a new command SetKeyspaceDurabilityPolicy has been added, which allows users to change the durability policy of an existing keyspace.

    If semi-sync is not being used then durability policy should be set to none for the keyspace. This is also the default option.

    If semi-sync is being used then durability policy should be set to semi_sync for the keyspace and --enable_semi_sync should be set on vttablets.

    VTOrc - Deprecation of Durability Configuration

    The Durability configuration is deprecated and removed from VTOrc. Instead VTOrc will find the durability policy of the keyspace from the topology server. This allows VTOrc to monitor and repair multiple keyspaces which have different durability policies in use.

    VTOrc will ignore keyspaces which have no durability policy specified in the keyspace record. This is to avoid clobbering an existing config from a previous release. So on upgrading to v14, users must run the command SetKeyspaceDurabilityPolicy specified above, to ensure that VTOrc continues to work as desired. The recommended upgrade path is to upgrade VTCtld, run SetKeyspaceDurabilityPolicy and then upgrade VTOrc.

    Advisory locking optimizations

    Work has gone into making the advisory locks (get_lock(), release_lock(), etc.) release reserved connections faster and in more situations than before.

    Pre-Legacy Resharding is now deprecated

    A long time ago, the sharding column and type were specified at the keyspace level. This syntax is now deprecated and will be removed in v15.


    The entire changelog for this release can be found here.

    The release includes 1101 commits (excluding merges)

    Thanks to all our contributors: @FancyFane, @GuptaManan100, @Juneezee, @K-Kumar-01, @Phanatic, @ajm188, @akenneth, @aquarapid, @arthurschreiber, @brendar, @cuishuang, @dasl-, @dbussink, @deepthi, @dependabot[bot], @derekperkins, @doeg, @fatih, @frouioui, @harshit-gangal, @malpani, @matthiasr, @mattlord, @mattrobenolt, @notfelineit, @pjambet, @rohit-nayak-ps, @rsajwani, @shlomi-noach, @simon-engledew, @systay, @utk9, @vmg, @vmogilev, @y5w, @yields

    Source code(tar.gz)
    Source code(zip)
    vitess-14.0.0-9665c18.tar.gz(488.77 MB)
    vitess-14.0.0-9665c18.x86_64.rpm(487.00 MB)
    vitess_14.0.0-9665c18_amd64.deb(488.76 MB)
  • v14.0.0-rc1(Jun 7, 2022)

    Release of Vitess v14.0.0-RC1

    Major Changes

    Command-line syntax deprecations

    Vitess has begun a transition to a new library for CLI flag parsing. In order to facilitate a smooth transition, certain syntaxes that will not be supported in the future now issue deprecation warnings when used.

    The messages you will likely see, along with explanations and migrations, are:

    "Use of single-dash long flags is deprecated"

    Single-dash usage will be only possible for short flags (e.g. -v is okay, but -verbose is not).

    To migrate, update your CLI scripts from:

    $ vttablet -tablet_alias zone1-100 -init_keyspace mykeyspace ... # old way
    

    To:

    $ vttablet --tablet_alias zone1-100 --init_keyspace mykeyspace ... # new way
    

    "Detected a dashed argument after a position argument."

    As the full deprecation text goes on to (attempt to) explain, mixing flags and positional arguments will change in a future version that will break scripts.

    Currently, when invoking a binary like

    $ vtctl --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
    

    everything after the AddCellInfo is treated by package flag as a positional argument, and we then use a sub FlagSet to parse flags specific to the subcommand. So, at the top-level, flag.Args() returns ["AddCellInfo", "--root", "/vitess/global"].

    The library we are transitioning to is more flexible, allowing flags and positional arguments to be interwoven on the command-line. For the above example, this means that we would attempt to parse --root as a top-level flag for the vtctl binary. This will cause the program to exit on error, because that flag is only defined on the AddCellInfo subcommand.

    In order to transition, a standalone double-dash (literally, --) will cause the new flag library to treat everything following that as a positional argument, and also works with the current flag parsing code we use.

    So, to transition the above example without breakage, update the command to:

    $ vtctl --topo_implementation etcd2 AddCellInfo -- --root "/vitess/global"
    $ # the following will also work
    $ vtctl --topo_implementation etcd2 -- AddCellInfo --root "/vitess/global"
    $ # the following will NOT work, because --topo_implementation is a top-level flag, not a sub-command flag
    $ vtctl -- --topo_implementation etcd2 AddCellInfo --root "/vitess/global"
    

    New command line flags and behavior

    vttablet --heartbeat_on_demand_duration

    --heartbeat_on_demand_duration joins the already existing heartbeat flags --heartbeat_enable and --heartbeat_interval and adds new behavior to heartbeat writes.

    --heartbeat_on_demand_duration takes a duration value, such as 5s.

    The default value for --heartbeat_on_demand_duration is zero, which means the flag is not set and there is no change in behavior.

    When --heartbeat_on_demand_duration has a positive value, then heartbeats are only injected on demand, per internal requests. For example, when --heartbeat_on_demand_duration=5s, the tablet starts without injecting heartbeats. An internal module, like the lag throttle, may request the heartbeat writer for heartbeats. Starting at that point in time, and for the duration (a lease) of 5s in our example, the tablet will write heartbeats. If no other requests come in during that duration, then the tablet then ceases to write heartbeats. If more requests for heartbeats come while heartbeats are being written, then the tablet extends the lease for the next 5s following up each request. Thus, it stops writing heartbeats 5s after the last request is received.

    The heartbeats are generated according to --heartbeat_interval.

    Deprecation of --online_ddl_check_interval

    The flag --online_ddl_check_interval is deprecated and will be removed in v15. It has been unused in v13.

    Deprecation of --planner-version for vtexplain

    The flag --planner-version is deprecated and will be removed in v15. Instead, please use --planer_version.

    Online DDL changes

    Online DDL is generally available

    Online DDL is no longer experimental (with the exception of pt-osc strategy). Specifically:

    • Managed schema changes, the scheduler, the backing tables
    • Supporting SQL syntax
    • vitess strategy (online DDL via VReplication)
    • gh-ost strategy (online DDL via 3rd party gh-ost)
    • Recoverable migrations
    • Revertible migrations
    • Declarative migrations
    • Postponed migrations
    • and all other functionality

    Are all considered production-ready.

    pt-osc strategy (online DDL via 3rd party pt-online-schema-change) remains experimental.

    Throttling

    See new SQL syntax for controlling/viewing throttling for Online DDL, down below.

    ddl_strategy: 'vitess'

    --ddl_strategy now takes the value of vitess to indicate VReplication-based migrations. It is a synonym to online and uses the exact same functionality. In the future, the online term will phase out, and vitess will remain the term of preference.

    Example:

    vtctlclient ApplySchema -- --skip_preflight --ddl_strategy='vitess' --sql "alter table my_table add column my_val int not null default 0" commerce
    

    --singleton-context and REVERT migrations

    It is now possible to submit a migration with --singleton-context strategy flag, while there's a pending (queued or running) REVERT migration that does not have a --singleton-context flag.

    Support for CHECK constraints

    Online DDL operations are more aware of CHECK constraints, and properly handle the limitation where a CHECK's name has to be unique in the schema. As opposed to letting MySQL choose arbitrary names for shadow table's CHECK consraints, Online DDL now generates unique yet deterministic names, such that all shards converge onto same names.

    Online DDL attempts to preserve the original check's name as a suffix to the generated name, where possible (names are limited to 64 characters).

    Behavior changes

    • vtctlclient ApplySchema -- --uuid_list='...' now rejects a migration if an existing migration has the same UUID but with different migration_context.

    Table lifecycle

    Views

    Table lifecycle now supports views. It ensures to not purge rows from views, and does not keep views in EVAC state (they are immediately transitioned to DROP state).

    Fast drops

    On Mysql 8.0.23 or later, the states PURGE and EVAC are automatically skipped, thanks to 8.0.23 improvement to DROP TABLE speed of operation.

    Tablet throttler

    API changes

    Added /throttler/throttled-apps endpoint, which reports back all current throttling instructions. Note, this only reports explicit throttling requests (sych as ones submitted by /throtler/throttle-app?app=...). It does not list incidental rejections based on throttle thresholds.

    API endpoint /throttler/throttle-app now accepts a ratio query argument, a floating point in the range [0..1], where:

    • 0 means "do not throttle at all"
    • 1 means "always throttle"
    • any numbr in between is allowd. For example, 0.3 means "throttle in 0.3 probability", ie on a per request and based on a dice roll, there's a 30% change a request is denied. Overall we can expect about 30% of requests to be denied. Example: /throttler/throttle-app?app=vreplication&ratio=0.25

    See new SQL syntax for controlling/viewing throttling, down below.

    New Syntax

    Control and view Online DDL throttling

    We introduce the following syntax, to:

    • Start/stop throttling for all Online DDL migrations, in general
    • Start/stop throttling for a particular Online DDL migration
    • View throttler state
    ALTER VITESS_MIGRATION '<uuid>' THROTTLE [EXPIRE '<duration>'] [RATIO <ratio>];
    ALTER VITESS_MIGRATION THROTTLE ALL [EXPIRE '<duration>'] [RATIO <ratio>];
    ALTER VITESS_MIGRATION '<uuid>' UNTHROTTLE;
    ALTER VITESS_MIGRATION UNTHROTTLE ALL;
    SHOW VITESS_THROTTLED_APPS;
    

    default duration is "infinite" (set as 100 years)

    • allowed units are (s)ec, (m)in, (h)our ratio is in the range [0..1].
    • 1 means full throttle - the app will not make any progress
    • 0 means no throttling at all
    • 0.8 means on 8 out of 10 checks the app makes, it gets refused

    The syntax SHOW VITESS_THROTTLED_APPS is a generic call to the throttler, and returns information about all throttled apps, not specific to migrations

    SHOW VITESS_MIGRATIONS ... output now includes user_throttle_ratio

    This column is updated "once in a while", while a migration is running. Normally this is once a minute, but can be more frequent. The migration reports back what was the throttling instruction set by the user while it was/is running. This column does not indicate any actual lag-based throttling that takes place per production state. It only reports the explicit throttling value set by the user.

    Heartbeat

    The throttler now checks in with the heartbeat writer to request heartbeats, any time it (the throttler) is asked for a check.

    When --heartbeat_on_demand_duration is not set, there is no change in behavior.

    When --heartbeat_on_demand_duration is set to a positive value, then the throttler ensures that the heartbeat writer generated heartbeats for at least the following duration. This also means at the first throttler check, it's possible that heartbeats are idle, and so the first check will fail. As heartbeats start running, followup checks will get a more accurate lag evaluation and will respond accordingly. In a sense, it's a "cold engine" scenario, where the engine takes time to start up, and then runs smoothly.

    VDiff2

    We introduced a new version of VDiff -- currently marked as Experimental -- that executes the VDiff on tablets rather than in vtctld. While this is experimental we encourage you to try it out and provide feedback! This input will be invaluable as we improve this feature on the march toward a production-ready version. You can try it out by adding the --v2 flag to your VDiff command. Here's an example:

    $ vtctlclient --server=localhost:15999 VDiff -- --v2 customer.commerce2customer
    VDiff bf9dfc5f-e5e6-11ec-823d-0aa62e50dd24 scheduled on target shards, use show to view progress
    
    $ vtctlclient --server=localhost:15999 VDiff -- --v2 customer.commerce2customer show last
    
    VDiff Summary for customer.commerce2customer (bf9dfc5f-e5e6-11ec-823d-0aa62e50dd24)
    State: completed
    HasMismatch: false
    
    Use "--format=json" for more detailed output.
    
    $ vtctlclient --server=localhost:15999 VDiff -- --v2 --format=json customer.commerce2customer show last
    {
            "Workflow": "commerce2customer",
            "Keyspace": "customer",
            "State": "completed",
            "UUID": "bf9dfc5f-e5e6-11ec-823d-0aa62e50dd24",
            "HasMismatch": false,
            "Shards": "0"
    }
    

    :information_source: NOTE: even before it's marked as production-ready (feature complete and tested widely in 1+ releases), it should be safe to use and is likely to provide much better results for very large tables.

    For additional details, please see the RFC and the README.

    Durability Policy

    Deprecation of durability_policy Flag

    The durability policy for a keyspace is now stored in the keyspace record in the topo server. The durability_policy flag used by vtctl, vtctld, and vtworker binaries has been deprecated.

    New and Augmented Commands

    The vtctld command CreateKeyspace has been augmented to take in an additional argument called durability-policy which will allow users to set the desired durability policy for a keyspace at creation time.

    For existing keyspaces, a new command SetKeyspaceDurabilityPolicy has been added, which allows users to change the durability policy of an existing keyspace.

    If semi-sync is not being used then durability policy should be set to none for the keyspace. This is also the default option.

    If semi-sync is being used then durability policy should be set to semi_sync for the keyspace and --enable_semi_sync should be set on vttablets.

    Deprecation of Durability Configuration

    The Durability configuration is deprecated and removed from VTOrc. Instead VTOrc will find the durability policy of the keyspace from the topo server. This allows VTOrc to monitor and repair multiple keyspaces which have different durability policies in use.

    VTOrc will ignore the keyspaces which have no durability policy specified in the keyspace record. So on upgrading to v14, users must run the command SetKeyspaceDurabilityPolicy specified above, to ensure VTOrc continues to work as desired. The recommended upgrade path is to upgrade vtctld, run SetKeyspaceDurabilityPolicy and then upgrade VTOrc.


    Changelog

    Announcement

    General

    • v14 release notes: Online DDL is production ready #10310

    Bug fixes

    Backup and Restore

    • Skip repl config in restore when active reparents disabled #9675

    Cluster management

    • [grpcshim] Fix select race and move to internal package #9116
    • Use AllPrivs user for table GC transitions #9690
    • [topo/helpers] Make CopyShardReplications idempotent #9849
    • Fix the race between PromoteReplica and replication manager tick #9859
    • Propagate error from 'updateLocked' to client #10181
    • Revert semi_sync change to reintroduce enable_semi_sync and remove durability_policy #10201
    • The flag --online_ddl_check_interval belongs in vtctld #10320
    • vt/wrangler: fix deleting primary tablet if Shard does not exist #10373
    • topo: cache CellInfo along with conn #10408

    General

    • Sanitize tx serializer log & error messages #9802
    • Changes for make tools #10117
    • Sanitize log messages #10367

    Query Serving

    • Extract collation data to enable distinct aggregation #9639
    • Add QueryRowsAffected and QueryRowsReturned to vttablet metrics and deprecate QueryRowCounts #9656
    • Use the gen4 planner for queries with outer joins #9657
    • Gen4: make sure to not merge unsharded tables from different keyspaces #9665
    • Parser fix for CREATE VIEW statement #9693
    • Add collation parsing to generated columns in Create Table #9694
    • Fixed missing order/group by, limit, having in derived #9701
    • Online DDL: resubmitting a migration with same UUID retries it in case it was cancelled or failed. #9704
    • Query plan/rules: apply for multi-table statement #9747
    • Stop rewriting JoinCondition with USING #9767
    • Manage MySQL Replication Status States Properly #9853
    • Fix __sq_has_values1 error with PulloutSubquery #9855
    • OnlineDDL executor: route all VReplicationExec through single function #9861
    • fix: planner panic on derived tables sorting in query builder #9869
    • feat: add weightstring for distinct #9874
    • Fix: Sequence query to ignore reserved and transaction #9968
    • fix: dual query with exists clause having system table query in it #9969
    • fix: make concatenate and limit concurrent safe #9979
    • ApplySchema: allow-zero-in-date embedded as query comment in call to ExecuteFetchAsDba #9998
    • Fix: reserved connection retry logic when vttablet or mysql is down #10005
    • gen4: Fix sub query planning when the outer query is a dual query #10007
    • Fix parsing of bind variables in a few places #10015
    • Route explain tab plan to the proper Keyspace #10027
    • OnlineDDL: double statement validation upon submission #10065
    • Fix Gen4 only_full_group_by regression #10069
    • Fix Gen4 group/order by with derived table #10074
    • Only start SQL thread temporarily to WaitForPosition if needed #10104
    • Vitess online ddl: modify a column from textual to non-textual, ignore charset #10116
    • Fix StreamExecute in Gen4CompareV3 #10122
    • Do not mutate replication state in WaitSourcePos and ignore tablets with SQL_Thread stopped in ERS #10148
    • Fix for empty results when no shards can be found to route to #10152
    • fix: handle reserved connection reset when tx killer has locked the connection #10153
    • Do not send field query when using reserved connection #10163
    • Backwards compatible replication status to state transition #10167
    • sqlparser: Handle case sensitive AST option for table options #10191
    • Emit the ENGINE field for table options as case sensitive #10197
    • check for connectionID before adding to querylist #10212
    • Fix handling of unsigned and zerofill in parser and normalization #10220
    • Fix parsing of the foreign key constraint actions in different order #10224
    • Fix handling of VISIBLE or INVISIBLE keyword on indexes #10243
    • Fix formatting for function expressions and booleans #10255
    • fix: concatenate engine primitive #10257
    • Move all schemadiff comparisons to canonical form #10261
    • Do not cache plans that are invalid because of --no_scatter #10279
    • Fix parsing the special case convert charset logic #10288
    • Deprecate flag --online_ddl_check_interval #10308
    • Fix schema tracking issue when PRIMARY tablet changes #10335
    • Remove normalization of integral length types #10336
    • Fix failure when Unowned lookup IS NULL #10346
    • Fix for empty results when no shards can be found to route to [v3] #10360
    • Revert super_read_only config file changes #10366
    • OnlineDDL/vitess fix: convert data type to JSON #10390
    • OnlineDDL/vrepl: more error codes, spatial types #10394
    • Online DDL fix: instant is possible for VIRTUAL, not STORED #10411
    • schemadiff: column name check in CHECK constraint is case insensitive #10413
    • making log sanitization more precise #10417
    • v3: Fix issue when no routes are found from a lookup #10422
    • Tablet throttler: dynamic ThrottleMetricThreshold #10439
    • Change use_super_read_only default back to false #10448

    VReplication

    • Support VDiff across DB versions #9679
    • Externalize Lookup VIndexes properly when not stopping after copy #9771
    • Add TEXT field comparison support to evalengine (for VDiff) #9790
    • Add CHAR field comparison support to evalengine (for VDiff) #9800
    • Check for nil vschema.Table in StreamMigrator #9828
    • VPlayer: use stored/binlogged ENUM index value in WHERE clauses #9868
    • vstreamer: flatten savepoint events #9892
    • vstreamer: do not forward savepoint events #9907
    • VReplication: use strict sql_mode for Online DDL #9963
    • VStreamer: recompute table plan if a new table is encountered for the same id #9978
    • Increase max vrepl workflow source definition size from 64KiB to 16MiB #10018
    • VReplication: maintain original column case for pkCols #10033
    • Avoid deadlocks related to 0 receiver behavior #10132
    • SwitchTraffic should switch everything when no tablet_types provided #10434

    VTAdmin

    • [vtadmin] threadsafe dynamic clusters #10044

    VTorc

    • BugFix: Resolve Recoveries at the end #10286

    vtexplain

    • Case-insensitive planner-version flag in VTExplain #10086
    • fix: check that all keyspaces loaded successfully before using them #10396

    CI/Build

    Build/CI

    • Track full git sha1 string instead of the short version #9970
    • Skip CI workflows based on the list of modified files #10031
    • CI: fix flaky test via extended timeout in vrepl_stress_suite #10231
    • Fix tabletmanager_throttler flakyness: increase wait time #10271
    • remove references to MySQL and Percona 5.6 #10295

    Cluster management

    • fix endtoend onlineddl flakyness caused by --heartbeat_on_demand_duration #10236

    General

    • Upgrade main to go1.18 #9893
    • Upgrade main to go1.18.1 #10101
    • Upgrade main to go1.18.3 #10447

    Query Serving

    • More testing for vreplication unique key violation protection #9988
    • Fix to --heartbeat_on_demand_duration race condition #10247
    • OnlineDDL: adding an endtoend test to validate partial shard REVERT #10338

    Dependabot

    Examples

    • build(deps): bump gopkg.in/yaml.v2 from 2.2.5 to 2.2.8 in /examples/are-you-alive #9429

    Documentation

    Documentation

    • Update release notes with web/vtctld2 deprecation announcement and flag. #9735
    • Add help text for LegacyVtctlCommand and VtctldCommand #9837

    Examples

    • Fix commands in readme files to use double dashes for arguments #10389

    Enhancement

    Build/CI

    • Check for proto drift in the CI #9644
    • make generate_ci_workflows: mysql80 workflows for selected tests #9740
    • Add vitess/lite ubi8 images for mysql80, including for arm64 #9830

    CLI

    • [vtctldclient] Add entrypoint for GetVersion rpc #9994

    Cluster management

    • Change semi-sync information to use the parameter passed and deprecate enable_semi_sync #9725
    • Filter candidates which cannot make forward progress after ERS #9765
    • [vtctldserver] Add locking checks to Delete{Keyspace,Shard} #9777
    • [topo] ShardReplicationFix typed responses #9876
    • tablet lag throttler: small API improvements #10045
    • OnlineDDL: skip GetSchema where possible #10107
    • Refresh ephemeral information before cluster operations in VTOrc #10115
    • On demand heartbeats via --heartbeat_on_demand_duration, used by the tablet throttler #10198
    • Adds DurabilityPolicy to the KeyspaceInfo in the topo and the associated RPCs #10221
    • Refactor Durability Policy implementation and usage to read the durability policy from the keyspace #10375
    • Use Durability Policy in the topo server in VTOrc and deprecate Durability config #10423

    Examples

    • Add consul-topo for local example #9806
    • Add vtadmin to local example by default #10430
    • Halve request and double limit for RAM in the example cluster #10450

    General

    • Binlog event parsing: better analysis ; support for semi-sync #9596
    • Support views in Online DDL #9606
    • [VEP-4, phase 1] Flag Deprecation Warnings #9733
    • Table lifecycle: support fast DROP TABLE introduced in MySQL 8.0.23 #9778
    • Flavor capabilities: MySQL GR and more #10451

    Query Serving

    • Harmonize error codes for pool timeouts expiring across all pools #9483
    • Store innodbRowsRead in a Counter instead of a Gauge #9609
    • Improve performance of information schema query. #9632
    • OnlineDDL: reject duplicate UUID with different migration_context #9637
    • Reduce the number of reserved connections when setting system variables #9641
    • OnlineDDL: 'vitess' strategy, synonym for 'online' #9642
    • Gen4: Rework how aggregation is planned #9643
    • Support for ALTER TABLE ... PARTITION BY ... #9683
    • OnlineDDL: force (once) vreplication's WithDDL to run before running a vitess migration #9702
    • Push projection to union #9703
    • Make gen4 the default planner for release 14 #9710
    • Cleanup of parsing of Partitions and additional parsing support #9712
    • Introducing schemadiff, a declarative diff for table/view CREATE statements #9719
    • Online DDL vitess migration's cut-over: query buffering #9755
    • Online DDL declarative migrations now use schemadiff, removing tengo #9772
    • Online DDL: ready_to_complete hint column #9813
    • Add parsing support for prepare statements #9818
    • Online DDL: vitess migrations cut-over to have zero race conditions #9832
    • Add parsing support for Trim grammar function #9834
    • Add parsing support for LATERAL keyword #9843
    • Add parsing support for JSON Utility Functions #9850
    • Remove the Gateway interface #9852
    • Gen4: Add UPDATE planning #9871
    • Add parsing support for JSON_TABLE() #9879
    • Add parsing support for JSON value creators functions #9880
    • fix: allow multiple distinct columns on single shards #9940
    • Add parsing support for Json schema validation functions #9971
    • Add parsing support for JSON Search Functions #9990
    • deps: upgrade grpc and protobuf #10024
    • feat: allow more complex expressions to be used in outer queries #10034
    • schemadiff: load, normalize, validate and compare schemas #10048
    • Add parsing support for JSON value modification and attribute functions #10062
    • Always reset reserved shard session on CODE_UNAVAILABLE #10063
    • Adds parsing for NOW in DEFAULT clause #10085
    • Gen4 feature: optimize OR queries to IN #10090
    • Relax singleton-context constraint for pending non-singleton-context reverts #10114
    • Add parsing support for Partition Definitions #10127
    • feat gen4: allow pushing aggregations inside derived tables #10128
    • schemadiff: support for RangeRotationStrategy diff hint #10131
    • Avoid enabling reserved connection on show query #10138
    • Add support for non aggregated columns in OrderedAggregate #10139
    • schemadiff: CanonicalStatementString(), utilize sqlparser.CanonicalString() #10142
    • feat: make gen4 plan subsharding queries better #10151
    • schemadiff: EntityDiff has a 'Entities() (from Entity, to Entity)' function #10161
    • Introducing a SQL syntax to control and view Online DDL throttling #10162
    • feat: optimize EXISTS queries through AST rewriting #10174
    • schemadiff: logically Apply() a diff to a CreateTable, CreateView or to a Schema #10183
    • Using LCS to achieve shortest column reordering list #10184
    • enhancement: handle advisory locks #10186
    • schemadiff: normalize() table. Set names to all keys #10188
    • schemadiff: validate() table structure at the end of apply() #10189
    • schemadiff: schema is immutable through Apply() #10190
    • VReplication: use PK equivalent (PKE) if no PK found #10192
    • Add parsing support for Window Functions #10199
    • schemadiff: normalize table options case #10200
    • Allow updating unowned lookup vindex columns. #10207
    • partial dml execution logic ehancement #10223
    • gen4: JOIN USING planning #10226
    • schemadiff: partitions validations #10229
    • Add parsing support for Subpartition Definitions #10232
    • schemadiff: analyze and apply ADD PARTITION and DROP PARTITION statements #10234
    • Add convenience functions for working with schemadiff objects #10238
    • Add parsing support for expressions in index definitions #10240
    • Update the parsing support for JSON_VALUE #10242
    • schemadiff: support functional indexes #10244
    • Fix formatting of character set annotation on string literals #10245
    • schemadiff: use ParseStrictDDL whenever expecting a DDL statement #10246
    • Fix parsing of encryption attribute for create database #10249
    • Add support for parsing MATCH in a foreign key definition #10250
    • Add parsing support for additional index and table options #10251
    • Handle additional DDL in column definitions #10253
    • evalengine: CASE expressions #10262
    • vt/sqlparser: support SRID #10267
    • Improve dealing with check constraints #10268
    • Add parsing and schemadiff support for ALTER TABLE ALTER CHECK #10269
    • Add unary utf8mb3 charset marker #10274
    • Allow updating the subsequent diff #10289
    • Improve the formatting for partitioning #10291
    • Add additional normalization rules for schemadiff #10296
    • schemadiff: RangeRotationDistinctStatements partitioning hint #10309
    • Remove internal savepoint post query execution #10311
    • feat: use column offsets when HAVING predicate has aggregation #10312
    • Improve parser and schemadiff support for ALTER TABLE #10313
    • Online DDL: fast (and correct) RANGE PARTITION operations, 1st iteration #10315
    • Add support for additional encoding configurations #10321
    • Add parsing support for Regular Expressions #10322
    • schemadiff: validate columns referenced by generated columns #10328
    • schemadiff: case insensitive col/index/partition name comparison #10330
    • [gen4] More aggregation support #10332
    • Add better int normalization #10340
    • Additional schemadiff improvements for indexes #10345
    • Structured schemadiff errors #10356
    • Add validation of check constraints #10357
    • schemadiff: validate columns referenced by FOREIGN KEY #10359
    • Aggregation on top of LIMIT #10361
    • Add support for parsing ARRAY in cast / convert operations #10362
    • Online DDL: support for CHECK CONSTRAINTS #10369
    • schemadiff: constraint name diff hints strategies #10380
    • Replace 'CREATE TABLE LIKE' with programmatic copy #10381
    • Streaming implementation of Projection engine primitive #10384
    • feat: do not use lookup vindexes for IS NULL predicates #10388
    • Online DDL: support INSTANT DDL special plan, flag protected (flag undocumented for now) #10402
    • feat: make v3 not plan IS NULL queries against a lookup vindex #10412
    • Online DDL: support Instant DDL for change of column's DEFAULT #10414
    • Parsing Support for XML functions #10438
    • Formalize MySQL capabilities by flavor/version #10445

    TabletManager

    • messaging: support vt_message_cols to limit cols #9670
    • Table lifecycle: support views #9776

    VReplication

    • OnlineDDL: delete _vt.vreplication entry as part of CLEANUP operation #9638
    • Tablet server/VReplication: use information_schema.columns to get the list of columns instead of "select * from table" #9794
    • Add --drop_constraints to MoveTables #9904
    • Support throttling vstreamer copy table work on source tablets #9923
    • Improve escape handling for vindex materializer #9929
    • VReplication: fail on unrecoverable errors #9973
    • Prefer using REPLICA tablets when selecting vreplication sources #10040
    • Improve Tablet Refresh Behavior in VReplication Traffic Switch Handling #10058
    • VReplication: use db_filtered user for vstreams #10080
    • VStream API: allow cells to be specified for picking source tablets to stream from #10294
    • MoveTables: adjust datetimes when importing from non-UTC sources into UTC targets #10102
    • Vdiff2: initial release #10382

    VTAdmin

    • [vtadmin-web] Add REACT_APP_READONLY_MODE flag #9731
    • [vtadmin] custom discovery resolver #9977
    • [vtadmin] grpc healthserver + channelz #10038
    • [Dynamic Clusters] No Duplicates #10043
    • [vtadmin] full dynamic cluster config #10071
    • [vtadmin] non-blocking resolver #10205
    • [vtadmin] tablet topopools + other cleanup #10325
    • [vtadmin] cluster Closer implementation + dynamic Cluster bugfix #10355

    VTCombo

    • Add flag to vtcombo to load a json-encoded test topology file #9633

    VTorc

    • Revocation phase using durability policies #9659
    • VTOrc: Refactor and reload of ephemeral information for remaining recovery functions #10150
    • vtorc: more auditing in GracefulPrimaryTakeover #10285

    vttestserver

    • vtcombo: Add the ability to specify routing rules in test topology #9695

    web UI

    • [vtctld] Add 'enable_vtctld_ui' flag to vtctld to explicitly enable (or disable) the vtctld2 UI #9614

    Feature Request

    Cluster management

    • [vtctldserver] Migrate Backup RPC to vtctldserver/client #9798
    • [vtctldclient] Add entrypoint for ApplySchema #9829
    • [vtctldserver] Migrate BackupShard RPC #9857
    • [vtctldserver] Migrate RemoveBackup RPC #9865
    • [vtctldserver] Migrate RestoreFromBackup RPC #9873
    • [vtctldserver] Migrate ShardReplicationFix RPC #9878
    • [vtctldserver] Migrate SourceShard{Add,Delete} RPCs #9886
    • [vtctldserver] ExecuteFetchAs{App,DBA} #9890
    • [vtctldserver] Migrate ShardReplication{Add,Remove} RPCs #9899
    • [vtctldserver] Migrate GetPermissions PR#9903 #10013

    Observability

    • [vtadmin] Add prom metrics endpoint support #10334

    Query Serving

    • mysql: allow parsing the 'info' field of MySQL packets #9651
    • Add support for insert using select #9748
    • Add parsing support for Multi INDEX HINTS #9811

    VTAdmin

    • [VTAdmin] Tablet Actions (vtctld2 Parity) #9601
    • [VTAdmin][vtctldserver] Keyspace Functions #9667
    • Allow --no-rbac flag that allows users to not pass rbac config #9972
    • [vtadmin] grpc dynamic clusters #10050
    • [vtadmin] schema cache #10120
    • [vtadmin] cell apis #10227
    • [vtadmin] reload schema #10300
    • [vtadmin] reparenting actions #10351

    Internal Cleanup

    Build/CI

    • Remove usage of additional uuid package #10307

    CLI

    • [vtctl] Deprecate query commands #9934
    • [vtctl] Deprecate Throttler RPCs #9962
    • [vtadmin] Remove deprecated dynamic cluster flag #10067
    • Update vtctl help output to use double dashes for long flag names #10405

    Cluster management

    • Move deprecation warning to just before we actually invoke the legacy command path #9787
    • Online DDL: cleanup old vtctld+topo flow #9816
    • Rename ApplySchemaRequest.request_context=>migration_context #9824
    • rename master_alias to primary_alias in reparent_journal table #10098
    • Use the HealthCheckImpl in vtctld instead of the LegacyHealthCheck #10254
    • Cleanup: removed legacy, unused code (online DDL paths in topo) #10379

    Examples

    • Examples: fix warnings #9875
    • Migrate remaining vtctldclient commands in local example #9894

    General

    • Support sanitizing potentially sensitive information in log messages #9550
    • [VEP-4, phase 1] Flag cleanup in go/cmd #9896

    Query Serving

    • Delete discovery gateway #9500
    • Remove queryserver-config-terse-errors impact on log messages #9634
    • Split aggregation in scalar and ordered aggregation #9810
    • Minor fix requested by linter #9989
    • remove set_var session field from proto #10135

    TabletManager

    • delete RPCs that were deprecated in 13.0 #10099

    VReplication

    • remove changeMasterToPrimary, we no longer need it #10100
    • Ensure all legacy sharding commands are clearly deprecated #10281

    VTAdmin

    • [vtadmin] threadsafe vtsql #10000
    • [vtsql] Remove vtsql.PingContext check from dial calls #10037
    • [vtadmin] Refactor api.getSchemas => cluster.GetSchemas #10108
    • [vtadmin] Remove explicit Dial-ing for vtctldclient proxy #10233
    • [vtadmin] Removing explicit Dial-ing for vtsql proxy #10237
    • [vtadmin] proper alias types for tablet RPCs #10290
    • [vtadmin] Remove mutex, now that vtexplain has no global state #10387
    • [vtadmin] Rename reparent-related RPCs, endpoints, methods, RBAC actions #10404

    vtctl

    • Fixes #8277: vtctld logs leak pii http headers #9669

    vtexplain

    • replace planner-version with planner_version in vtexplain #10420

    Performance

    Schema Tracker

    • Improve performance of the DetectSchemaChange query #10416

    Testing

    Build/CI

    • More aggressive tests for vitess migration cut-over #9956
    • fix tablegc flaky test #10230

    CLI

    • [VEP-4, part 1] Flag cleanup in go/mysql and some endtoend tests #9910
    • [VEP4, phase 1] Update flags in endtoend/recovery #9911
    • [VEP4, phase 1] Update flags in endtoend/schemadiff #9912
    • [VEP4, phase 1] Update flags and imports for endtoend/{sharded,sharding} #9913
    • [VEP 4, phase 1] Cleanup flags in endtoend/vault (and some comments in endtoend/tabletmanager) #9916
    • [VEP4, phase 1] fix flags in tests endtoend/versionupgrade #9917
    • [VEP4, phase 1] [endtoend/vreplication] update flags in tests #9918
    • [VEP4, phase 1] [endtoend/vtcombo] fix flags in tests #9919
    • [VEP4, phase1] Cleanup flags in endtoend/vtgate and endtoend/vtorc #9930
    • [VEP4, phase1] fix flags in tests in endtoend/worker and strings in go/trace #9931

    General

    • test: use T.TempDir to create temporary test directory #10433

    Query Serving

    • vtexplain: added multicol test to show subshard routing #9697
    • Compare Vitess against MySQL in E2E tests #9965
    • Make the tests explicit for the SET variables for MySQL 5.7 #10263

    The release includes 1065 commits (excluding merges)

    Thanks to all our contributors: @FancyFane, @GuptaManan100, @Juneezee, @K-Kumar-01, @Phanatic, @ajm188, @akenneth, @aquarapid, @arthurschreiber, @brendar, @cuishuang, @dasl-, @dbussink, @deepthi, @dependabot[bot], @derekperkins, @doeg, @fatih, @frouioui, @harshit-gangal, @malpani, @matthiasr, @mattlord, @mattrobenolt, @notfelineit, @pjambet, @rohit-nayak-ps, @rsajwani, @shlomi-noach, @simon-engledew, @systay, @utk9, @vmg, @vmogilev, @y5w, @yields

    Source code(tar.gz)
    Source code(zip)
    vitess-14.0.0-rc1-7821f05.tar.gz(487.90 MB)
    vitess-14.0.0_rc1-7821f05.x86_64.rpm(486.13 MB)
    vitess_14.0.0-rc1-7821f05_amd64.deb(487.91 MB)
  • v12.0.4(May 20, 2022)

    Release of Vitess v12.0.4

    Major Changes

    Cluster management

    A race condition #9819 that happens when running PlannedReparentShard was fixed through #9859.

    Query Serving

    Two major bugs on UNION are fixed via this release. The first one, #10257, solves an issue around the UNION executor, where the execution context was not handled properly in a concurrent environment. The second one, #9945, was detected by one of our UNION tests becoming flaky in the CI, it got solved by #9979, which focuses on improving the concurrency of UNION executions.

    A panic when ordering results in descending order on a hash vindex is also fixed. The original issue can be found here #10019.

    Changelog

    Bug fixes

    Cluster management

    • Fix the race between PromoteReplica and replication manager tick #9859

    Query Serving

    • Route explain table plan to the proper Keyspace #10028
    • Multiple fixes to UNION planning and execution #10344

    The release includes 5 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @frouioui, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.4-c462e63.tar.gz(400.65 MB)
    vitess-12.0.4-c462e63.x86_64.rpm(398.94 MB)
    vitess_12.0.4-c462e63_amd64.deb(400.65 MB)
  • v13.0.1(Apr 14, 2022)

    Release of Vitess v13.0.1

    Changelog

    Bug fixes

    Cluster management

    • Fix the race between PromoteReplica and replication manager tick #9859

    Query Serving

    • Fix __sq_has_values1 error with PulloutSubquery #9864
    • Fix planner panic on derived tables sorting in query builder #9959
    • Fix make concatenate and limit concurrent safe #9981
    • Fix reserved connection retry logic when vttablet or mysql is down #10005
    • Fix Gen4 sub query planning when the outer query is a dual query #10007
    • Fix parsing of bind variables in a few places #10015
    • Fix route explain tab plan to the proper Keyspace #10029
    • Fix Sequence query to ignore reserved and transaction #10054
    • Fix dual query with exists clause having system table query in it #10055
    • Fix Gen4 only_full_group_by regression #10079

    VReplication

    • VPlayer use stored/binlogged ENUM index value in WHERE clauses #9868

    CI/Build

    Security

    • Upgrade to go1.17.9 #10088 The go1.17.9 version, released 2022-04-12, includes security fixes to the crypto/elliptic and encoding/pem packages, as well as bug fixes to the linker and runtime). More information here.

    Enhancement

    Query Serving

    • Fix allow multiple distinct columns on single shards #10047
    • Fix add parsing for NOW in DEFAULT clause #10085

    The release includes 30 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @frouioui, @harshit-gangal, @mattlord, @rohit-nayak-ps, @systay, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-13.0.1-f4a77b6.tar.gz(463.28 MB)
    vitess-13.0.1-f4a77b6.x86_64.rpm(461.63 MB)
    vitess_13.0.1-f4a77b6_amd64.deb(463.29 MB)
  • v13.0.0(Feb 22, 2022)

    Release of Vitess v13.0.0

    Major Changes

    Vitess now has native support for MySQL collations

    When using the gen4 planner, Vitess is now capable of performing collation-aware string comparisons in the vtgates. This improves the performance and reliability of several query plans that were previously relying on a debug-only SQL API in MySQL to perform these comparisons remotely. It also enables new query plans that were previously not possible.

    A full list of the supported collations can be found in the Vitess documentation.

    The native Evaluation engine in vtgate has been greatly improved

    The SQL evaluation engine that runs inside the vtgates has been rewritten mostly from scratch to more closely match MySQL's behavior. This allows Vitess to execute more parts of the query plans locally, and increases the complexity and semantics of the SQL expressions that can be used to perform cross-shard queries.

    vttablet -use_super_read_only flag now defaults to true

    The default value used to be false. What this means is that during a failover, we will set super_read_only on database flavors that support them (MySQL 5.7+ and Percona 5.7+). In addition, all Vitess-managed databases will be started with super-read-only in the cnf file. It is expected that this change is safe and backwards-compatible. Anyone who is relying on the current behavior should pass -use_super_read_only=false on the vttablet command line, and make sure they are using a custom my.cnf instead of the one provided as the default by Vitess.

    vtgate -buffer_implementation now defaults to keyspace_events

    The default value used to be healthcheck. The new keyspace_events implementation has been tested in production with good results and shows more consistent buffering behavior during PlannedReparentShard operations. The keyspace_events implementation utilizes heuristics to detect additional cluster states where buffering is safe to perform, including cases where the primary may be down. If there is a need to revert back to the previous buffer implementation, ensure buffering is enabled in vtgate and pass the flag -buffer_implementation=healthcheck.

    ddl_strategy: -postpone-completion flag

    ddl_strategy (either @@ddl_strategy in VtGate or -ddl_strategy in vtctlclient ApplySchema) supports the flag -postpone-completion

    This flag indicates that the migration should not auto-complete. This applies for:

    • any CREATE TABLE
    • any DROP TABLE
    • ALTER table in online strategy
    • ALTER table in gh-ost strategy

    Note that this flag is not supported for pt-osc strategy.

    Behavior of migrations with this flag:

    • an ALTER table begins, runs, but does not cut-over.
    • CREATE or DROP migrations are silently not even scheduled

    alter vitess_migration ... cleanup

    A new query is supported:

    alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' cleanup
    

    This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As a reminder, once a migration's artifacts are cleaned up, the migration is no longer revertible.

    alter vitess_migration ... complete

    A new query is supported:

    alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete
    

    This command indicates that a migration executed with -postpone-completion is good to complete. Behavior:

    • For running ALTERs (online and gh-ost) which are ready to cut-over: cut-over imminently (though not immediately
      • cut-over depends on polling interval, replication lag, etc)
    • For running ALTERs (online and gh-ost) which are only partly through the migration: they will cut-over automatically when they complete their work, as if -postpone-completion wasn't indicated
    • For queued CREATE and DROP migrations: "unblock" them from being scheduled. They'll be scheduled at the scheduler' s discretion. there is no guarantee that they will be scheduled to run immediately.

    vtctl/vtctlclient ApplySchema: ALTER VITESS_MIGRATION

    vtctl ApplySchema now supports ALTER VITESS_MIGRATION ... statements. Example:

    $ vtctl ApplySchema -skip_preflight -sql "alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete" commerce
    

    vtctl/vtctlclient ApplySchema: allow zero in date

    vtctl/vtctlclient ApplySchema now respects -allow-zero-in-date for direct strategy. For example, the following statement is now accepted:

    vtctlclient ApplySchema -skip_preflight -ddl_strategy='direct -allow-zero-in-date' -sql "create table if not exists t2(id int primary key, dt datetime default '0000-00-00 00:00:00')" commerce
    

    vtctl/vtctlclient ApplySchema -uuid_list

    vtctlient ApplySchema now support a new optional -uuid_list flag. It is possible for the user to explicitly specify the UUIDs for given migration(s). UUIDs must be in a specific format. If given, number of UUIDs must match the number of DDL statements. Example:

    vtctlclient OnlineDDL ApplySchema -sql "drop table t1, drop table t2" -uuid_list "d08f0000_51c9_11ec_9cf2_0a43f95f28a3,d08f0001_51c9_11ec_9cf2_0a43f95f28a3" commerce
    

    Vitess will assign each migration with given UUID in order of appearance. It is the user's responsibility to ensure given UUIDs are globally unique. If the user submits a migration with an already existing UUID, that migration never gets scheduled nor executed.

    vtctl/vtctlclient ApplySchema -migration_context

    -migration_context flag is synonymous to -request_context. Either will work. We will encourage use of -migration_context as it is more consistent with output of SHOW VITESS_MIGRATIONS ... which includes the migration_context column.

    vtctl/vtctlclient OnlineDDL ... complete

    Complementing the alter vitess_migration ... complete query, a migration can also be completed via vtctl or vtctlclient:

    vtctlclient OnlineDDL <keyspace> complete <uuid>
    

    For example:

    vtctlclient OnlineDDL commerce complete d08ffe6b_51c9_11ec_9cf2_0a43f95f28a3
    

    vtctl/vtctlclient OnlineDDL -json

    The command now accepts an optional -json flag. With this flag, the output is a valid JSON listing all columns and rows.

    vtadmin-web updated to node v16.13.0 (LTS)

    Building vtadmin-web now requires node >= v16.13.0 (LTS). Upgrade notes are given in https://github.com/vitessio/vitess/pull/9136.

    PlannedReparentShard for cluster initialization

    For setting up the cluster and electing a primary for the first time, PlannedReparentShard should be used instead of InitShardPrimary.

    InitShardPrimary is a forceful command and copies over the executed gtid set from the new primary to all the other replicas. So, if the user isn't careful, it can lead to some replicas not being setup correctly and lead to errors in replication and recovery later. PlannedReparentShard is a safer alternative and does not change the executed gtid set on the replicas forcefully. It is the preferred alternate to initialize the cluster.

    If using a custom init_db.sql that omits SET sql_log_bin = 0, then InitShardPrimary should still be used instead of PlannedReparentShard.

    Durability Policy flag

    A new flag has been added to vtctl, vtctld and vtworker binaries which allows the users to set the durability policies.

    If semi-sync is not being used then -durability_policy should be set to none. This is also the default option.

    If semi-sync is being used then -durability_policy should be set to semi_sync and -enable_semi_sync should be set in vttablets.

    Incompatible Changes

    Error message change when vttablet row limit exceeded:

    • In previous Vitess versions, if the vttablet row limit (-queryserver-config-max-result-size) was exceeded, an error like:
      ERROR 10001 (HY000): target: unsharded.0.master: vttablet: rpc error: code = ResourceExhausted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      would be reported to the client.
      

      To avoid confusion, the error mapping has been changed to report the error as similar to:

      ERROR 10001 (HY000): target: unsharded.0.primary: vttablet: rpc error: code = Aborted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      instead
      
    • Because of this error code change, the vttablet metric: vttablet_errors{error_code="ABORTED"} will be incremented upon this type of error, instead of the previous metric: vttablet_errors{error_code="RESOURCE_EXHAUSTED"}
    • In addition, the vttablet error message logged is now different. Previously, a (misleading; due to the PoolFull) error was logged:
      E0112 09:48:25.420641  278125 tabletserver.go:1368] PoolFull: Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      

      Post-change, a more accurate warning is logged instead:

      W0112 09:38:59.169264   35943 tabletserver.go:1503] Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      
    • If you were using -queryserver-config-terse-errors to redact some log messages containing bind vars in 13.0-SNAPSHOT, you should now instead enable -sanitize_log_messages which sanitizes all log messages containing sensitive info

    Column types for textual queries now match MySQL's behavior

    The column types for certain queries performed on a vtgate (most notably, those that SELECT system variables) have been changed to match the types that would be returned if querying a MySQL instance directly: textual fields that were previously returned as VARBINARY will now appear as VARCHAR.

    This change should not have an impact on MySQL clients/connectors for statically typed programming languages, as these clients coerce the returned rows into whatever types the user has requested, but clients for dynamic programming languages may now start returning as "string" values that were previously returned as "bytes".

    Deprecations

    vtgate -gateway_implementation flag is deprecated (and ignored)

    Support for discoverygateway is being dropped. tabletgateway is now the only supported implementation. Scripts using this flag should be updated to remove the flag as it will be deleted in the next release.

    web/vtctld2 is deprecated and can optionally be turned off

    The vtctld2 web interface is no longer maintained and is planned for removal in Vitess 16. Motivation for this change and a roadmap to removing the web/vtctld2 codebase is given in https://github.com/vitessio/vitess/issues/9686.

    Vitess operators can optionally disable the vtctld2 web interface ahead of time by calling the vtctld process with the flag enable_vtctld_ui=false. For more details on this flag, see https://github.com/vitessio/vitess/pull/9614.


    Changelog

    Bug fixes

    Backup and Restore

    • vtctl: Check ShardReplicationStatuses error and make sure they are well returned #8966
    • Skip replication config in restore when active reparents are disabled #9688
    • Fix semi-sync for Backup and RestoreFromBackup commands by calling SetReplicationSource RPC #9730

    Cluster management

    • Fix vtctlclient DeleteCellInfo(force=true) command with downed local topo #9081
    • Fix race condition when rapidly starting/stopping health check #9152
    • Topo write error in change tablet type #9157
    • Default build info tablet tags to off #9187
    • Replication manager should not start health ticks after calling Promote Replica #9274
    • Fix error checking in ERS #9330
    • Fix for context cancellation in ERS #9503

    General

    • Avoid some pollution of CLI argument namespace across vtgate and vttablet #8931
    • Update to the latest Tengo version #9110

    Orchestrator

    • Set orc maintenance mode on tablet that is being promoted by PRS #8859

    Query Serving

    • Fix for schema tracker not recognizing RENAME for schema reloading #8963
    • Handle sql_mode differently for new value change #9014
    • Fix savepoint support with reserved connections #9030
    • Use a DBA user pool for the heartbeat writer #9033
    • Handle single column vindex correctly with multi-columns in Gen4 #9060
    • Remove tablet healthcheck cache record on error #9106
    • Fix request buffering while reparenting #9112
    • Ensure that the hex query predicates are normalized for planner cache #9118
    • Fix table/sizes query, to allow vttablet schema engine to see partitioned tables #9188
    • Remove keyspace from the query before sending it to the shards #9190
    • Make sure to copy bindvars when using them concurrently #9191
    • Fix owned vindex query to keep alias in table expression #9244
    • Use decoded hex string when calculating the keyspace ID #9277
    • Fix show statement in prepare call #9280
    • Avoid parenthesis around default null in BLOB, TEXT, GEOMETRY, and JSON column #9301
    • Allow unowned lookup_unique vindex columns to be null #9302
    • Estimate replica lag when the number of seconds behind from mysqld is unknown #9308
    • Clear existing keyspace schema in vtgate before [re]loading it #9437
    • Fix AST copy issue in the Gen4Fallback planner #9487
    • Fix a bug in the Gen4 planner for join query pushing output column on the left-hand side #9521
    • Extract collation data to enable distinct aggregation #9645
    • Use the gen4 planner for queries with outer joins #9671
    • Gen4: make sure not to merge unsharded tables from different keyspaces #9676
    • Fixed missing order/group by, limit, having in derived #9707

    TabletManager

    • Prevent Race Conditions Between Tablet Deletes and Updates #9237

    VReplication

    • Fix boolean parameter order in DropSources call for v2 flows #9175
    • VReplication Workflow Create: Check for Copying state while waiting for streams to start #9206
    • MoveTables: update vschema while moving tables with autoincrement from sharded to unsharded #9288
    • Schema Engine now accountable for Online DDL swapped tables #9324
    • Take MySQL Column Type Into Account in VStreamer #9331
    • Allow source columns to be specified multiple times in a vreplication rule filter #9335
    • Fixed case-sensitive TabletType in SwitchTraffic #9440
    • Use LOCK TABLES during SwitchWrites to prevent lost writes #9481
    • VReplication Workflow: manage SQL_MODE as MySQL does in mysqldump #9505
    • Routing Rules: Improved escaping of reserved and invalid names for tables and keyspaces #9522
    • Ignore internal tables in MoveTables and Reshard Workflows #8992,#9578

    VTCombo

    • vtcombo: CREATE DATABASE creates shards as specified by the topology #9130

    CI/Build

    Build/CI

    • Remove Percona5.6 Unit Tests for Vitess 13.0 #9067
    • Addition of upgrade downgrade testing to the test suite #9300, #9473, #9501
    • Re-enable default Make behavior of failing target if any command fails #9319
    • ubuntu-latest now uses MySQL 8.0.26, let us override it with latest 8.0.x #9368
    • Add vtadmin binary to release and Docker images #9405

    VTAdmin

    • Enforce node + npm versions with .npmrc #9336

    vtexplain

    • Correct handling of column types for queries involving multiple tables #9260

    Dependabot

    Build/CI

    • Upgrade go-proxyproto to v0.6.1 to fix vulnerabilities #9425

    Java

    • Bump log4j-core from 2.15.0 to 2.16.0 in /java #9387
    • Bump log4j-api from 2.16.0 to 2.17.1 in /java #9423
    • Updates all major Java dependencies to latest #9599

    VTAdmin

    • Bump tmpl from 1.0.4 to 1.0.5 in /web/vtadmin #9278

    Enhancement

    Authn/z

    • Add support for TLS certification revocation list (CRL) files #8807

    Backup and Restore

    • Set super_read_only off during restore (achieves same as PR #8929) #9240

    Cluster management

    • Support -allow-zero-in-date in ApplySchema 'direct' strategy #9273
    • Remove legacy OnlineDDL code deprecated in earlier version #8971
    • Add version-info to tablet tags in the topo #8973
    • Add default collation to VTGate and VTTablet #9097
    • Use relay log position in PRS while choosing the new primary #9270
    • vtctl OnlineDDL supports 'complete' command #9298
    • vtctl ApplySchema accepts 'ALTER VITESS_MIGRATION...' statements #9303
    • Make the heartbeat writer use 2 pools #9320
    • New vtctl ApplySchema flag -uuid_list #9325
    • Fix ERS to be used for initialization as well #9511
    • Fix ERS to work when the primary candidate's replication is stopped #9512
    • Add 'enable_vtctld_ui' flag to vtctld to explicitly enable (or disable) the vtctld2 UI (backport) #9713

    Examples

    • Update operator.yaml to v2.5.0 #9016

    General

    • vitessdriver: add support for DistributedTx #9451

    Observability

    • Modify terse errors to also redact errors in logs #9094

    Query Serving

    • Initial implementation of partitions (Create statement) #8691
    • Add parsing for common table expressions using WITH clause #8918
    • Support SQL SELECT LIMIT #8944
    • Support for union in subquery in Gen4 #8948
    • Support for ordering using derived table columns in Gen4 #8961
    • Add planner-version flag to vtexplain #8979
    • Rewrite NOT expressions for cleaner plans #8987
    • Minimize logging of errors when loading keyspace with disabled schema tracking #8989
    • Support for ordering on vindex function and straight_join queries to be planned by ignoring the hint as warning in Gen4 #8990
    • Improve internal subquery dependencies update in Gen4 #8998
    • Added tokenizer support for NCHAR_STRING #9006
    • Query timeout in comment directive in Gen4 #9008
    • Added brackets in default value if column is of type TEXT, BLOB, GEOMETRY or JSON #9011
    • Addition of the filter primitive and planning in Gen4 #9017
    • Initial integration of the collation module in Gen4 #9018
    • Support extract function in the SQL parser #9029
    • Add collation name to the type definition in Gen4 #9038
    • Support semi-join in Gen4 #9039
    • Changed Collations and Expression grammar #9075
    • Support transactions with OLAP workload #9115
    • Use of advisory lock in prepare queries #9129
    • Add hash join primitive and planning in Gen4 #9140
    • Respect --allow-zero-in-date in CREATE and in declarative migrations #9142
    • Improve merging of DBA queries in Gen4 #9183
    • Evaluate REVERTibility of a migration in OnlineDDL #9232
    • Improve group by queries planning when they use a sharding key #9243
    • Make sure VTGate and VTTablet collations are matching #9248
    • Support filtering on derived union system table with star projection #9263
    • MySQL: Better handling of query failures within a transaction to match the MySQL behavior #9269
    • Add Tuples and InOp implementation to EvalEngine #9281
    • Use evalengine for Vindex lookups in route primitive #9291
    • Adds parsing support for string introducers #9309
    • Multi-column vindex support for DML #9338
    • Add list (IN()) support to vindex functions #9426
    • Fix parsing of table names that start with a number #9456
    • Make the MaxWaiters setting configurable for the OLAP pool #9484
    • Add support for health service in gRPC server #9528
    • Filter after left join #9531
    • Add Information_Schema to evalengine #9536
    • Faster and safer unsharded query planning #9542
    • Add support for PLANNER SQL comment #9545
    • MySQL: use UTF8MB4 consistently as the connection charset #9558
    • Do not auto retry after "server lost" errors (errno 2013) in vttablet #9009
    • Online DDL: migration with same context and SQL now considered duplicate and silently marked as complete #9107
    • Support ALTER VITESS_MIGRATION ... CLEANUP statement #9160
    • OnlineDDL: implementing -postpone-completion, ALTER VITESS_MIGRATION ... COMPLETE #9171
    • OnlineDDL: support concurrent REVERT migrations #9192
    • VTGate: change default query buffer implementation (used to avoid client errors during state transitions such as PlannedReparentShard) #9359 #9360
    • Fix misleading error mapping for row limit error #9448
    • Add -sanitize_log_messages tablet flag to sanitize all log messages which may contain sensitive info and remove -queryserver-config-terse-errors impact on log messages #9636

    TabletManager

    • Have vttablet use super_read_only by default when a tablet becomes a replica #9312
    • MySQL: Add ConnectionReady callback to Handler interface #9496
    • MySQL: Add UnimplementedHandler struct #9526

    VReplication

    • Support VStream with keyspaces_to_watch #8988
    • VEvent: Add Keyspace/Shard properties to proto #9386
    • Initial refactoring of the 'vstream * from' functionality in vtgate #9392
    • Add flag to keep routing rules in v2 vrepl workflows #9441
    • SwitchTraffic: check vreplication lag before switching #9538

    VTAdmin

    • [vtadmin] viper config support #9154
    • [VTAdmin] Update react-scripts, postcss #9493

    VTorc

    • Have a common EmergencyReparentShard (ERS) implementation for both VtOrc and Vtctl #8492
    • Use PlannedReparentShard (PRS) in VtOrc while doing Graceful Primary Takeover #9258
    • Make PRS use Durability Rules #9259
    • Use PRS in Vtorc for electing new primary #9409

    vtctl

    • ListAllTablets: improve efficiency within vitess (fewer topo calls) and for caller (keyspace and tablet type filters) #9560
    • vtctl OnlineDDL show: export all columns, support -json #9568
    • vtctl ApplySchema: introduce -migration_context flag, synonym to -request_context #9572

    vttestserver

    • Add option to run vtcombo and vttest local cluster with a real topo server #9176

    Feature Request

    Query Serving

    • Add support for SET statements in OLAP mode #9253
    • Multi-column vindex support in select statements #9326
    • Multi-column vindex support in update and delete queries #9467
    • Multicolumn partial vindex selection #9390
    • Subsharding vindex support #9428
    • Non-unique vindex routing in update and delete query #9554

    VTAdmin

    • [vtadmin] Add create+delete shards functionality #9012
    • Add experimental whoami api endpoint #9233
    • [vtadmin] Add ping feature, Dropdown and Dialog components #9318
    • [VTAdmin] Dynamic Cluster support #9544

    VTorc

    • Merge InitShardPrimary functionality into PlannedReparentShard #9276

    Internal Cleanup

    Backup and Restore

    • Inclusive naming: use new RPCs in vtctl/vtctld/wrangler etc. #9181

    Cluster management

    • reparent_journal: add backwards compatible alter statement #9439
    • [vtctl] command deprecations #8967
    • [wrangler] Cleanup duplicate wrangler methods #9015
    • Migrate k8s topo CRD to v1 api #9045
    • [schematools] Move more functions out of wrangler to package importable by grpcvtctldserver #9123
    • [vtctld] Migrate ReloadSchema* rpcs #8832
    • [vtctldserver] migrate ExecuteHook #9024
    • [vtctldclient-codegen] Add support in codegen for streaming RPCs #9064

    General

    • Delete unused go/cmd/automation_{client,server} binaries #9234
    • Inclusive naming: replace master with primary #9427,#9430,#9438,#9454,#9103,#9182
    • naming: delete old code that was needed for version compatibility #9516
    • MySQL: Pass mysql.Conn through {Hash,PlainText,Caching} storage interfaces #9264

    Java

    • build(deps): bump log4j-api and log4j-core from 2.13.3 to 2.15.0 in /java #9348,#9349

    Observability

    • Changed master to primary everywhere #9506

    Query Serving

    • Deprecate and ignore gateway_implementation flag to vtgate #9482
    • Deprecate vtctl command IgnoreHealthError #9594

    VReplication

    • Fixed legacy vreplication data that may be using master tablet types #9497

    VTAdmin

    • [vtadmin-web] Upgrade to node v16.13.0 LTS #9136

    VTorc

    • Fix linting issues in Vtorc #9449
    • Pass Semi sync information from durability policy and use it to log discrepancies #9533

    vtctl

    • Inclusive naming: removing vtctl flags and commands with master in them that were deprecated in 12.0 #9179
    • vtctl: delete deprecated blacklisted_tables flag #9406

    web UI

    • Migrate jQuery (pt. 1) #9239
    • Migrate jQuery (pt. 2/3) #9256

    Performance

    Query Serving

    • vttablet: Better use of vtprotobuf memory pool #9365

    Testing

    Query Serving

    • Fuzzing: Add more fuzzers #9249

    The release includes 2020 commits (excluding merges)

    Thanks to all our contributors: @AdamKorcz, @FancyFane, @GuptaManan100, @Phanatic, @Thirumalai-Shaktivel, @ajm188, @aquarapid, @arthurschreiber, @arvind-murty, @askdba, @carsonoid, @chapsuk, @choo-stripe, @dasl-, @dbussink, @dctrwatson, @deepthi, @dependabot[bot], @derekperkins, @doeg, @frouioui, @hallaroo, @harshit-gangal, @hkdsun, @king-11, @mattlord, @mattrobenolt, @mvh-stripe, @notfelineit, @oscerd, @pH14, @ritwizsinha, @rohit-nayak-ps, @shichao-an, @shlomi-noach, @systay, @utk9, @vmg, @y5w, @zhongr3n

    Source code(tar.gz)
    Source code(zip)
    vitess-13.0.0-bc4a960.tar.gz(463.06 MB)
    vitess-13.0.0-bc4a960.x86_64.rpm(461.37 MB)
    vitess_13.0.0-bc4a960_amd64.deb(463.07 MB)
  • v13.0.0-rc1(Feb 1, 2022)

    Release of Vitess v13.0.0-rc1

    Major Changes

    Vitess now has native support for MySQL collations

    When using the gen4 planner, Vitess is now capable of performing collation-aware string comparisons in the vtgates. This improves the performance and reliability of several query plans that were previously relying on a debug-only SQL API in MySQL to perform these comparisons remotely. It also enables new query plans that were previously not possible.

    A full list of the supported collations can be found in the Vitess documentation.

    The native Evaluation engine in vtgate has been greatly improved

    The SQL evaluation engine that runs inside the vtgates has been rewritten mostly from scratch to more closely match MySQL's behavior. This allows Vitess to execute more parts of the query plans locally, and increases the complexity and semantics of the SQL expressions that can be used to perform cross-shard queries.

    vttablet -use_super_read_only flag now defaults to true

    The default value used to be false. What this means is that during a failover, we will set super_read_only on database flavors that support them (MySQL 5.7+ and Percona 5.7+). In addition, all Vitess-managed databases will be started with super-read-only in the cnf file. It is expected that this change is safe and backwards-compatible. Anyone who is relying on the current behavior should pass -use_super_read_only=false on the vttablet command line, and make sure they are using a custom my.cnf instead of the one provided as the default by Vitess.

    vtgate -buffer_implementation now defaults to keyspace_events

    The default value used to be healthcheck. The new keyspace_events implementation has been tested in production with good results and shows more consistent buffering behavior during PlannedReparentShard operations. The keyspace_events implementation utilizes heuristics to detect additional cluster states where buffering is safe to perform, including cases where the primary may be down. If there is a need to revert back to the previous buffer implementation, ensure buffering is enabled in vtgate and pass the flag -buffer_implementation=healthcheck.

    ddl_strategy: -postpone-completion flag

    ddl_strategy (either @@ddl_strategy in VtGate or -ddl_strategy in vtctlclient ApplySchema) supports the flag -postpone-completion

    This flag indicates that the migration should not auto-complete. This applies for:

    • any CREATE TABLE
    • any DROP TABLE
    • ALTER table in online strategy
    • ALTER table in gh-ost strategy

    Note that this flag is not supported for pt-osc strategy.

    Behavior of migrations with this flag:

    • an ALTER table begins, runs, but does not cut-over.
    • CREATE or DROP migrations are silently not even scheduled

    alter vitess_migration ... cleanup

    A new query is supported:

    alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' cleanup
    

    This query tells Vitess that a migration's artifacts are good to be cleaned up asap. This allows Vitess to free disk resources sooner. As a reminder, once a migration's artifacts are cleaned up, the migration is no longer revertible.

    alter vitess_migration ... complete

    A new query is supported:

    alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete
    

    This command indicates that a migration executed with -postpone-completion is good to complete. Behavior:

    • For running ALTERs (online and gh-ost) which are ready to cut-over: cut-over imminently (though not immediately
      • cut-over depends on polling interval, replication lag, etc)
    • For running ALTERs (online and gh-ost) which are only partly through the migration: they will cut-over automatically when they complete their work, as if -postpone-completion wasn't indicated
    • For queued CREATE and DROP migrations: "unblock" them from being scheduled. They'll be scheduled at the scheduler' s discretion. there is no guarantee that they will be scheduled to run immediately.

    vtctl/vtctlclient ApplySchema: ALTER VITESS_MIGRATION

    vtctl ApplySchema now supports ALTER VITESS_MIGRATION ... statements. Example:

    $ vtctl ApplySchema -skip_preflight -sql "alter vitess_migration '9748c3b7_7fdb_11eb_ac2c_f875a4d24e90' complete" commerce
    

    vtctl/vtctlclient ApplySchema: allow zero in date

    vtctl/vtctlclient ApplySchema now respects -allow-zero-in-date for direct strategy. For example, the following statement is now accepted:

    vtctlclient ApplySchema -skip_preflight -ddl_strategy='direct -allow-zero-in-date' -sql "create table if not exists t2(id int primary key, dt datetime default '0000-00-00 00:00:00')" commerce
    

    vtctl/vtctlclient ApplySchema -uuid_list

    vtctlient ApplySchema now support a new optional -uuid_list flag. It is possible for the user to explicitly specify the UUIDs for given migration(s). UUIDs must be in a specific format. If given, number of UUIDs must match the number of DDL statements. Example:

    vtctlclient OnlineDDL ApplySchema -sql "drop table t1, drop table t2" -uuid_list "d08f0000_51c9_11ec_9cf2_0a43f95f28a3,d08f0001_51c9_11ec_9cf2_0a43f95f28a3" commerce
    

    Vitess will assign each migration with given UUID in order of appearance. It is the user's responsibility to ensure given UUIDs are globally unique. If the user submits a migration with an already existing UUID, that migration never gets scheduled nor executed.

    vtctl/vtctlclient ApplySchema -migration_context

    -migration_context flag is synonymous to -request_context. Either will work. We will encourage use of -migration_context as it is more consistent with output of SHOW VITESS_MIGRATIONS ... which includes the migration_context column.

    vtctl/vtctlclient OnlineDDL ... complete

    Complementing the alter vitess_migration ... complete query, a migration can also be completed via vtctl or vtctlclient:

    vtctlclient OnlineDDL <keyspace> complete <uuid>
    

    For example:

    vtctlclient OnlineDDL commerce complete d08ffe6b_51c9_11ec_9cf2_0a43f95f28a3
    

    vtctl/vtctlclient OnlineDDL -json

    The command now accepts an optional -json flag. With this flag, the output is a valid JSON listing all columns and rows.

    vtadmin-web updated to node v16.13.0 (LTS)

    Building vtadmin-web now requires node >= v16.13.0 (LTS). Upgrade notes are given in https://github.com/vitessio/vitess/pull/9136.

    PlannedReparentShard for cluster initialization

    For setting up the cluster and electing a primary for the first time, PlannedReparentShard should be used instead of InitShardPrimary.

    InitShardPrimary is a forceful command and copies over the executed gtid set from the new primary to all the other replicas. So, if the user isn't careful, it can lead to some replicas not being setup correctly and lead to errors in replication and recovery later. PlannedReparentShard is a safer alternative and does not change the executed gtid set on the replicas forcefully. It is the preferred alternate to initialize the cluster.

    If using a custom init_db.sql that omits SET sql_log_bin = 0, then InitShardPrimary should still be used instead of PlannedReparentShard.

    Durability Policy flag

    A new flag has been added to vtctl, vtctld and vtworker binaries which allows the users to set the durability policies.

    If semi-sync is not being used then -durability_policy should be set to none. This is also the default option.

    If semi-sync is being used then -durability_policy should be set to semi_sync and -enable_semi_sync should be set in vttablets.

    Incompatible Changes

    Error message change when vttablet row limit exceeded:

    • In previous Vitess versions, if the vttablet row limit (-queryserver-config-max-result-size) was exceeded, an error like:
      ERROR 10001 (HY000): target: unsharded.0.master: vttablet: rpc error: code = ResourceExhausted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      would be reported to the client.
      

      To avoid confusion, the error mapping has been changed to report the error as similar to:

      ERROR 10001 (HY000): target: unsharded.0.primary: vttablet: rpc error: code = Aborted desc = Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      instead
      
    • Because of this error code change, the vttablet metric: vttablet_errors{error_code="ABORTED"} will be incremented upon this type of error, instead of the previous metric: vttablet_errors{error_code="RESOURCE_EXHAUSTED"}
    • In addition, the vttablet error message logged is now different. Previously, a (misleading; due to the PoolFull) error was logged:
      E0112 09:48:25.420641  278125 tabletserver.go:1368] PoolFull: Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      

      Post-change, a more accurate warning is logged instead:

      W0112 09:38:59.169264   35943 tabletserver.go:1503] Row count exceeded 10000 (errno 10001) (sqlstate HY000) ...
      

    Column types for textual queries now match MySQL's behavior

    The column types for certain queries performed on a vtgate (most notably, those that SELECT system variables) have been changed to match the types that would be returned if querying a MySQL instance directly: textual fields that were previously returned as VARBINARY will now appear as VARCHAR.

    This change should not have an impact on MySQL clients/connectors for statically typed programming languages, as these clients coerce the returned rows into whatever types the user has requested, but clients for dynamic programming languages may now start returning as "string" values that were previously returned as "bytes".

    Deprecations

    vtgate -gateway_implementation flag is deprecated (and ignored)

    Support for discoverygateway is being dropped. tabletgateway is now the only supported implementation. Scripts using this flag should be updated to remove the flag as it will be deleted in the next release.


    Changelog

    Bug fixes

    Backup and Restore

    • vtctl: Check ShardReplicationStatuses error and make sure they are well returned #8966

    Cluster management

    • Fix vtctlclient DeleteCellInfo(force=true) command with downed local topo #9081
    • Fix race condition when rapidly starting/stopping health check #9152
    • Topo write error in change tablet type #9157
    • Default build info tablet tags to off #9187
    • Replication manager should not start health ticks after calling Promote Replica #9274
    • Fix error checking in ERS #9330
    • Fix for context cancellation in ERS #9503

    General

    • Avoid some pollution of CLI argument namespace across vtgate and vttablet #8931
    • Update to the latest Tengo version #9110

    Orchestrator

    • Set orc maintenance mode on tablet that is being promoted by PRS #8859

    Query Serving

    • Fix for schema tracker not recognizing RENAME for schema reloading #8963
    • Handle sql_mode differently for new value change #9014
    • Fix savepoint support with reserved connections #9030
    • Use a DBA user pool for the heartbeat writer #9033
    • Handle single column vindex correctly with multi-columns in Gen4 #9060
    • Remove tablet healthcheck cache record on error #9106
    • Fix request buffering while reparenting #9112
    • Ensure that the hex query predicates are normalized for planner cache #9118
    • Fix table/sizes query, to allow vttablet schema engine to see partitioned tables #9188
    • Remove keyspace from the query before sending it to the shards #9190
    • Make sure to copy bindvars when using them concurrently #9191
    • Fix owned vindex query to keep alias in table expression #9244
    • Use decoded hex string when calculating the keyspace ID #9277
    • Fix show statement in prepare call #9280
    • Avoid parenthesis around default null in BLOB, TEXT, GEOMETRY, and JSON column #9301
    • Allow unowned lookup_unique vindex columns to be null #9302
    • Estimate replica lag when the number of seconds behind from mysqld is unknown #9308
    • Clear existing keyspace schema in vtgate before [re]loading it #9437
    • Fix AST copy issue in the Gen4Fallback planner #9487
    • Fix a bug in the Gen4 planner for join query pushing output column on the left-hand side #9521

    TabletManager

    • Prevent Race Conditions Between Tablet Deletes and Updates #9237

    VReplication

    • Fix boolean parameter order in DropSources call for v2 flows #9175
    • VReplication Workflow Create: Check for Copying state while waiting for streams to start #9206
    • MoveTables: update vschema while moving tables with autoincrement from sharded to unsharded #9288
    • Schema Engine now accountable for Online DDL swapped tables #9324
    • Take MySQL Column Type Into Account in VStreamer #9331
    • Allow source columns to be specified multiple times in a vreplication rule filter #9335
    • Fixed case-sensitive TabletType in SwitchTraffic #9440
    • Use LOCK TABLES during SwitchWrites to prevent lost writes #9481
    • VReplication Workflow: manage SQL_MODE as MySQL does in mysqldump #9505
    • Routing Rules: Improved escaping of reserved and invalid names for tables and keyspaces #9522
    • Ignore internal tables in MoveTables and Reshard Workflows #8992,#9578

    VTCombo

    • vtcombo: CREATE DATABASE creates shards as specified by the topology #9130

    CI/Build

    Build/CI

    • Remove Percona5.6 Unit Tests for Vitess 13.0 #9067
    • Addition of upgrade downgrade testing to the test suite #9300, #9473, #9501
    • Re-enable default Make behavior of failing target if any command fails #9319
    • ubuntu-latest now uses MySQL 8.0.26, let us override it with latest 8.0.x #9368
    • Add vtadmin binary to release and Docker images #9405

    VTAdmin

    • Enforce node + npm versions with .npmrc #9336

    vtexplain

    • Correct handling of column types for queries involving multiple tables #9260

    Dependabot

    Build/CI

    • Upgrade go-proxyproto to v0.6.1 to fix vulnerabilities #9425

    Java

    • Bump log4j-core from 2.15.0 to 2.16.0 in /java #9387
    • Bump log4j-api from 2.16.0 to 2.17.1 in /java #9423
    • Updates all major Java dependencies to latest #9599

    VTAdmin

    • Bump tmpl from 1.0.4 to 1.0.5 in /web/vtadmin #9278

    Enhancement

    Authn/z

    • Add support for TLS certification revocation list (CRL) files #8807

    Backup and Restore

    • Set super_read_only off during restore (achieves same as PR #8929) #9240

    Cluster management

    • Support -allow-zero-in-date in ApplySchema 'direct' strategy #9273
    • Remove legacy OnlineDDL code deprecated in earlier version #8971
    • Add version-info to tablet tags in the topo #8973
    • Add default collation to VTGate and VTTablet #9097
    • Use relay log position in PRS while choosing the new primary #9270
    • vtctl OnlineDDL supports 'complete' command #9298
    • vtctl ApplySchema accepts 'ALTER VITESS_MIGRATION...' statements #9303
    • Make the heartbeat writer use 2 pools #9320
    • New vtctl ApplySchema flag -uuid_list #9325
    • Fix ERS to be used for initialization as well #9511
    • Fix ERS to work when the primary candidate's replication is stopped #9512

    Examples

    • Update operator.yaml to v2.5.0 #9016

    General

    • vitessdriver: add support for DistributedTx #9451

    Observability

    • Modify terse errors to also redact errors in logs #9094

    Query Serving

    • Initial implementation of partitions (Create statement) #8691
    • Add parsing for common table expressions using WITH clause #8918
    • Support SQL SELECT LIMIT #8944
    • Support for union in subquery in Gen4 #8948
    • Support for ordering using derived table columns in Gen4 #8961
    • Add planner-version flag to vtexplain #8979
    • Rewrite NOT expressions for cleaner plans #8987
    • Minimize logging of errors when loading keyspace with disabled schema tracking #8989
    • Support for ordering on vindex function and straight_join queries to be planned by ignoring the hint as warning in Gen4 #8990
    • Improve internal subquery dependencies update in Gen4 #8998
    • Added tokenizer support for NCHAR_STRING #9006
    • Query timeout in comment directive in Gen4 #9008
    • Added brackets in default value if column is of type TEXT, BLOB, GEOMETRY or JSON #9011
    • Addition of the filter primitive and planning in Gen4 #9017
    • Initial integration of the collation module in Gen4 #9018
    • Support extract function in the SQL parser #9029
    • Add collation name to the type definition in Gen4 #9038
    • Support semi-join in Gen4 #9039
    • Changed Collations and Expression grammar #9075
    • Support transactions with OLAP workload #9115
    • Use of advisory lock in prepare queries #9129
    • Add hash join primitive and planning in Gen4 #9140
    • Respect --allow-zero-in-date in CREATE and in declarative migrations #9142
    • Improve merging of DBA queries in Gen4 #9183
    • Evaluate REVERTibility of a migration in OnlineDDL #9232
    • Improve group by queries planning when they use a sharding key #9243
    • Make sure VTGate and VTTablet collations are matching #9248
    • Support filtering on derived union system table with star projection #9263
    • MySQL: Better handling of query failures within a transaction to match the MySQL behavior #9269
    • Add Tuples and InOp implementation to EvalEngine #9281
    • Use evalengine for Vindex lookups in route primitive #9291
    • Adds parsing support for string introducers #9309
    • Multi-column vindex support for DML #9338
    • Add list (IN()) support to vindex functions #9426
    • Fix parsing of table names that start with a number #9456
    • Make the MaxWaiters setting configurable for the OLAP pool #9484
    • Add support for health service in gRPC server #9528
    • Filter after left join #9531
    • Add Information_Schema to evalengine #9536
    • Faster and safer unsharded query planning #9542
    • Add support for PLANNER SQL comment #9545
    • MySQL: use UTF8MB4 consistently as the connection charset #9558
    • Do not auto retry after "server lost" errors (errno 2013) in vttablet #9009
    • Online DDL: migration with same context and SQL now considered duplicate and silently marked as complete #9107
    • Support ALTER VITESS_MIGRATION ... CLEANUP statement #9160
    • OnlineDDL: implementing -postpone-completion, ALTER VITESS_MIGRATION ... COMPLETE #9171
    • OnlineDDL: support concurrent REVERT migrations #9192
    • VTGate: change default query buffer implementation (used to avoid client errors during state transitions such as PlannedReparentShard) #9359 #9360
    • Fix misleading error mapping for row limit error #9448

    TabletManager

    • Have vttablet use super_read_only by default when a tablet becomes a replica #9312
    • MySQL: Add ConnectionReady callback to Handler interface #9496
    • MySQL: Add UnimplementedHandler struct #9526

    VReplication

    • Support VStream with keyspaces_to_watch #8988
    • VEvent: Add Keyspace/Shard properties to proto #9386
    • Initial refactoring of the 'vstream * from' functionality in vtgate #9392
    • Add flag to keep routing rules in v2 vrepl workflows #9441
    • SwitchTraffic: check vreplication lag before switching #9538

    VTAdmin

    • [vtadmin] viper config support #9154
    • [VTAdmin] Update react-scripts, postcss #9493

    VTorc

    • Have a common EmergencyReparentShard (ERS) implementation for both VtOrc and Vtctl #8492
    • Use PlannedReparentShard (PRS) in VtOrc while doing Graceful Primary Takeover #9258
    • Make PRS use Durability Rules #9259
    • Use PRS in Vtorc for electing new primary #9409

    vtctl

    • ListAllTablets: improve efficiency within vitess (fewer topo calls) and for caller (keyspace and tablet type filters) #9560
    • vtctl OnlineDDL show: export all columns, support -json #9568
    • vtctl ApplySchema: introduce -migration_context flag, synonym to -request_context #9572

    vttestserver

    • Add option to run vtcombo and vttest local cluster with a real topo server #9176

    Feature Request

    Query Serving

    • Add support for SET statements in OLAP mode #9253
    • Multi-column vindex support in select statements #9326
    • Multi-column vindex support in update and delete queries #9467
    • Multicolumn partial vindex selection #9390
    • Subsharding vindex support #9428
    • Non-unique vindex routing in update and delete query #9554

    VTAdmin

    • [vtadmin] Add create+delete shards functionality #9012
    • Add experimental whoami api endpoint #9233
    • [vtadmin] Add ping feature, Dropdown and Dialog components #9318
    • [VTAdmin] Dynamic Cluster support #9544

    VTorc

    • Merge InitShardPrimary functionality into PlannedReparentShard #9276

    Internal Cleanup

    Backup and Restore

    • Inclusive naming: use new RPCs in vtctl/vtctld/wrangler etc. #9181

    Cluster management

    • reparent_journal: add backwards compatible alter statement #9439
    • [vtctl] command deprecations #8967
    • [wrangler] Cleanup duplicate wrangler methods #9015
    • Migrate k8s topo CRD to v1 api #9045
    • [schematools] Move more functions out of wrangler to package importable by grpcvtctldserver #9123
    • [vtctld] Migrate ReloadSchema* rpcs #8832
    • [vtctldserver] migrate ExecuteHook #9024
    • [vtctldclient-codegen] Add support in codegen for streaming RPCs #9064

    General

    • Delete unused go/cmd/automation_{client,server} binaries #9234
    • Inclusive naming: replace master with primary #9427,#9430,#9438,#9454,#9103,#9182
    • naming: delete old code that was needed for version compatibility #9516
    • MySQL: Pass mysql.Conn through {Hash,PlainText,Caching} storage interfaces #9264

    Java

    • build(deps): bump log4j-api and log4j-core from 2.13.3 to 2.15.0 in /java #9348,#9349

    Observability

    • Changed master to primary everywhere #9506

    Query Serving

    • Deprecate and ignore gateway_implementation flag to vtgate #9482
    • Deprecate vtctl command IgnoreHealthError #9594

    VReplication

    • Fixed legacy vreplication data that may be using master tablet types #9497

    VTAdmin

    • [vtadmin-web] Upgrade to node v16.13.0 LTS #9136

    VTorc

    • Fix linting issues in Vtorc #9449
    • Pass Semi sync information from durability policy and use it to log discrepancies #9533

    vtctl

    • Inclusive naming: removing vtctl flags and commands with master in them that were deprecated in 12.0 #9179
    • vtctl: delete deprecated blacklisted_tables flag #9406

    web UI

    • Migrate jQuery (pt. 1) #9239
    • Migrate jQuery (pt. 2/3) #9256

    Performance

    Query Serving

    • vttablet: Better use of vtprotobuf memory pool #9365

    Testing

    Query Serving

    • Fuzzing: Add more fuzzers #9249

    The release includes 1963 commits (excluding merges)

    Thanks to all our contributors: @AdamKorcz, @FancyFane, @GuptaManan100, @Phanatic, @Thirumalai-Shaktivel, @ajm188, @aquarapid, @arthurschreiber, @arvind-murty, @askdba, @carsonoid, @chapsuk, @choo-stripe, @dasl-, @dbussink, @dctrwatson, @deepthi, @dependabot[bot], @derekperkins, @doeg, @frouioui, @hallaroo, @harshit-gangal, @hkdsun, @king-11, @mattlord, @mattrobenolt, @mvh-stripe, @notfelineit, @oscerd, @pH14, @ritwizsinha, @rohit-nayak-ps, @shichao-an, @shlomi-noach, @systay, @vmg, @y5w, @zhongr3n

    Source code(tar.gz)
    Source code(zip)
    vitess-13.0.0-rc1-3fa0fba.tar.gz(463.01 MB)
    vitess-13.0.0_rc1-3fa0fba.x86_64.rpm(461.35 MB)
    vitess_13.0.0-rc1-3fa0fba_amd64.deb(463.01 MB)
  • v12.0.3(Jan 10, 2022)

    Release of Vitess v12.0.3

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44832) (#9465).


    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-api from 2.16.0 to 2.17.1 in /java #9465

    The release includes 4 commits (excluding merges)

    Thanks to all our contributors: @dbussink, @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.3-cbc6ab6.tar.gz(400.48 MB)
    vitess-12.0.3-cbc6ab6.x86_64.rpm(398.75 MB)
    vitess_12.0.3-cbc6ab6_amd64.deb(400.49 MB)
  • v11.0.4(Jan 10, 2022)

    Release of Vitess v11.0.4

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44832) (#9464).

    Known Issues

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-api from 2.16.0 to 2.17.1 in /java #9464

    The release includes 6 commits (excluding merges)

    Thanks to all our contributors: @dbussink, @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.4-cb142ee.tar.gz(397.17 MB)
    vitess-11.0.4-cb142ee.x86_64.rpm(394.91 MB)
    vitess_11.0.4-cb142ee_amd64.deb(397.17 MB)
  • v10.0.5(Jan 10, 2022)

    Release of Vitess v10.0.5

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44832) (#9463).

    Known Issues

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-api from 2.16.0 to 2.17.1 in /java #9463

    The release includes 7 commits (excluding merges)

    Thanks to all our contributors: @dbussink, @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.5-dec816c.tar.gz(387.07 MB)
    vitess-10.0.5-dec816c.x86_64.rpm(384.80 MB)
    vitess_10.0.5-dec816c_amd64.deb(387.08 MB)
  • v12.0.2(Dec 17, 2021)

    Release of Vitess v12.0.2

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-45046) (#9396).

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs (CVE-2021-45046 and CVE-2021-44832) followed. These have been fixed in release 2.17.1. This release of Vitess, v12.0.2, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v12.0.3 instead, to benefit from the vulnerability patches.

    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-core from 2.15.0 to 2.16.0 in /java #9396

    The release includes 2 commits (excluding merges)

    Thanks to all our contributors: @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.2-d620dab.tar.gz(400.48 MB)
    vitess-12.0.2-d620dab.x86_64.rpm(398.75 MB)
    vitess_12.0.2-d620dab_amd64.deb(400.49 MB)
  • v11.0.3(Dec 17, 2021)

    Release of Vitess v11.0.3

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-45046) (#9395).

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v11.0.3, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v11.0.4 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.


    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-core from 2.15.0 to 2.16.0 in /java #9395

    Documentation

    Examples

    • change operator example to use v11.0.3 docker images #9403

    The release includes 3 commits (excluding merges) Thanks to all our contributors: @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.3-2412024.tar.gz(397.20 MB)
    vitess-11.0.3-2412024.x86_64.rpm(394.93 MB)
    vitess_11.0.3-2412024_amd64.deb(397.20 MB)
  • v10.0.4(Dec 17, 2021)

    Release of Vitess v10.0.4

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-45046) (#9394).

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v10.0.4, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v10.0.5 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.


    Changelog

    Dependabot

    Java

    • build(deps): bump log4j-core from 2.15.0 to 2.16.0 in /java #9394

    Documentation

    Examples

    • change operator example to use v10.0.4 docker images #9402

    The release includes 2 commits (excluding merges) Thanks to all our contributors: @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.4-23c5cf9.tar.gz(387.07 MB)
    vitess-10.0.4-23c5cf9.x86_64.rpm(384.80 MB)
    vitess_10.0.4-23c5cf9_amd64.deb(387.08 MB)
  • v12.0.1(Dec 14, 2021)

    Release of Vitess v12.0.1

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44228) (#9357), along with a few bug fixes.

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v12.0.1, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v12.0.3 instead, to benefit from the vulnerability patches.

    Changelog

    Bug fixes

    Query Serving

    • Ensure that hex query predicates are normalized for planner cache #9145
    • Gen4: Fail cross-shard join query with aggregation and grouping #9167
    • Make sure to copy bindvars when using them concurrently #9246
    • Remove keyspace from query before sending it on #9247
    • Use decoded hex string when calculating the keyspace ID #9293

    VReplication

    • Fix boolean parameter order in DropSources call for v2 flows #9178
    • Take MySQL Column Type Into Account in VStreamer #9355

    Cluster management

    • Restoring 'vtctl VExec' command #9227
      • This change restores vtctl VExec functionality. It was removed based on the assumption the only uses for this command were for Online DDL command. This was wrong, and VExec is also used as a wrapper around VReplication.

    CI/Build

    Build/CI

    • CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x #9373

    Internal Cleanup

    Java

    • build(deps): bump log4j-api from 2.13.3 to 2.15.0 in /java #9357

    The release includes 21 commits (excluding merges)

    Thanks to all our contributors: @GuptaManan100, @askdba, @deepthi, @dependabot[bot], @frouioui, @hallaroo, @harshit-gangal, @mattlord, @rohit-nayak-ps, @shlomi-noach, @systay, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.1-558a5eb.tar.gz(400.46 MB)
    vitess-12.0.1-558a5eb.x86_64.rpm(398.75 MB)
    vitess_12.0.1-558a5eb_amd64.deb(400.47 MB)
  • v11.0.2(Dec 14, 2021)

    Release of Vitess v11.0.2

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44228) (#9364), along with a few bug fixes.

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v11.0.2, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v11.0.4 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.


    Changelog

    Bug fixes

    VReplication

    • Fix how we identify MySQL generated columns #8796

    CI/Build

    Build/CI

    • CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x #9374

    Internal Cleanup

    Java

    • build(deps): bump log4j-api from 2.13.3 to 2.15.0 in /java #9364

    The release includes 7 commits (excluding merges)

    Thanks to all our contributors: @askdba, @deepthi, @systay, @tokikanno

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.2-4f80587.tar.gz(397.20 MB)
    vitess-11.0.2-4f80587.x86_64.rpm(394.93 MB)
    vitess_11.0.2-4f80587_amd64.deb(397.20 MB)
  • v10.0.3(Dec 14, 2021)

    Release of Vitess v10.0.3

    Announcement

    This patch is providing an update regarding the Apache Log4j security vulnerability (CVE-2021-44228) (#9363).

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v10.0.3, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v10.0.5 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.


    Changelog

    CI/Build

    Build/CI

    CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x #9375

    Internal Cleanup

    Java

    • build(deps): bump log4j-api from 2.13.3 to 2.15.0 in /java #9363

    The release includes 5 commits (excluding merges)

    Thanks to all our contributors: @deepthi, @frouioui

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.3-a267978.tar.gz(387.07 MB)
    vitess-10.0.3-a267978.x86_64.rpm(384.80 MB)
    vitess_10.0.3-a267978_amd64.deb(387.08 MB)
  • v12.0.0(Oct 26, 2021)

    Release of Vitess v12.0.0

    Major Changes

    This release includes the following major changes or new features.

    Inclusive Naming

    A number of CLI commands and vttablet RPCs have been deprecated as we move from master to primary. All functionality is backwards compatible except as noted under Incompatible Changes. Deprecated commands and flags will be removed in the next release (13.0). Deprecated vttablet RPCs will be removed in the subsequent release (14.0).

    Gen4 Planner

    The newest version of the query planner, Gen4, becomes an experimental feature as part of this release. While Gen4 has been under development for a few release cycles, we have now reached parity with its predecessor, v3.

    To use Gen4, VTGate's -planner_version flag needs to be set to Gen4Fallback.

    Query Buffering during failovers

    In order to support buffering during resharding cutovers in addition to primary failovers, a new implementation of query buffering has been added. This is considered experimental. To enable it the flag buffer_implementation should be set to keyspace_events. The existing implementation (flag value healthcheck) will be deprecated in a future release.

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v12.0.0, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v12.0.3 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174. This issue is fixed in release >= v12.0.1.

    Incompatible Changes

    CLI command output

    Wherever CLI commands produced master or MASTER for tablet type, they now produce primary or PRIMARY. Scripts and tools that depend on parsing command output will need to change.

    Example:

    $ vtctlclient -server localhost:15999 ListAllTablets 
    zone1-0000000100 sourcekeyspace 0 primary 192.168.0.134:15100 192.168.0.134:17100 [] 2021-09-24T01:12:00Z
    zone1-0000000101 sourcekeyspace 0 rdonly 192.168.0.134:15101 192.168.0.134:17101 [] <null>
    zone1-0000000102 sourcekeyspace 0 replica 192.168.0.134:15102 192.168.0.134:17102 [] <null>
    zone1-0000000103 sourcekeyspace 0 rdonly 192.168.0.134:15103 192.168.0.134:17103 [] <null>
    

    Default Behaviour Changes

    Enabling reserved connections

    Use of reserved connections is controlled by the vtgate flag -enable_system_settings. This flag has in the past defaulted to false (or OFF) in release versions (i.e. x.0 and x.0.y) of Vitess, and to true (or ON) in development versions.

    From Vitess 12.0.0 onwards, it defaults to true in all release and development versions. You can read more about this here. Hence you should specify this flag explicitly, so that you are sure whether it is enabled or not, regardless of which Vitess release/build/version you are running.

    If you have reserved connections disabled, you will get the old Vitess behavior: where most system settings (e.g. sql_mode) are just silently ignored by Vitess. In situations where you know your backend MySQL defaults are acceptable, this may be the correct setting to ensure the best possible performance of the vttablet <-> MySQL connection pools. This comes down to a trade-off between compatibility and performance/scalability. You should also review this section of the documentation when deciding whether or not to enable reserved connections.

    Deprecations

    CLI commands

    VExec is deprecated and removed. All Online DDL commands should be run through OnlineDDL.

    OnlineDDL revert is deprecated. Use REVERT VITESS_MIGRATION '...' SQL command either via ApplySchema or via vtgate.

    InitShardMaster is deprecated, use InitShardPrimary instead.

    SetShardIsMasterServing is deprecated, use SetShardIsPrimaryServing instead.

    Various command flags have been deprecated and new variants provided.

    • DeleteTablet flag allow_master replaced with allow_primary
    • PlannedReparentShard flag avoid_master replaced with avoid_tablet
    • PlannedReparentShard flag new_master replaced with new_primary
    • BackupShard flag allow_master replaced with allow_primary
    • Backup flag allow_master replaced with allow_primary
    • EmergencyReparentShard flag new_master replaced with new_primary
    • ReloadSchemeShard flag include_master replaced with include_primary
    • ReloadSchemaKeyspace flag include_master replaced with include_primary
    • ValidateSchemaKeyspace flag skip-no-master replaced with skip-no-primary

    Minor Changes

    Query Serving

    • Add SHOW VITESS_REPLICATION_STATUS Query Support #8900
      • The new SHOW VITESS_REPLICATION_STATUS command/query shows the MySQL replica/replication (not vreplication) health for the vitess deployment.

    Governance

    • Added @GuptaManan100 to maintainers #8833

    Changelog

    Bug fixes

    Build/CI

    • Update golang/x dependencies #8688
    • tests: use AtomicInt32 instead of int to fix races #8696

    Cluster management

    • Use consul lock properly #8310
    • Trivial: mysqlctl reinit_config -h was not showing help #8369
    • Port #8422 to main branch #8745
    • [vtctl] Add missing call to flag.Parse in commandGetRoutingRules #8795
    • Fix for padding in OrderAndCheckPartitions #8873
    • tabletmanager: correct implementations of deprecated RPCs for backwards compatibility #9059

    General

    • Avoid some pollution of CLI argument namespace across vtgate and vttablet #9026

    Observability

    • schema engine: reset table size stats when a table is dropped #8634

    Query Serving

    • Fix for function calls in DEFAULT value of CREATE TABLE statement in main #8477
    • Fixing multiple issues related to onlineddl/lifecycle #8500
    • boolean values should not be parenthesised in default clause #8502
    • mysql/server: abort connection if secure transport is required #8504
    • Clear only the current shard's schemacopy rows during schema reload #8519
    • go/mysql/auth: fix error message for wrong auth method. #8525
    • Fix handshake protocol with MySQL client #8537
    • Copy parameter data from byte buffer in COM_STMT_SEND_LONG_DATA #8562
    • Negative float default #8587
    • [vtexplain] prevent panic on show tables/show full tables #8601
    • OnlineDDL: better scheduling/cancellation logic #8603
    • Fix parsing of FIRST clause in ALTER TABLE #8614
    • onlineddl Executor: build schema with DBA user #8624
    • Handle subquery merging with references correctly #8662
    • Fixing error on deletes from owning table for a lookup being populated by CreateLookupVindex after a SwitchWrites #8701
    • Fixing a panic in vtgate with OLAP mode #8722
    • default to primary tablet if not set in VStream api #8755
    • Fix bug in parsing of version comments #8770
    • Proper merge of the SysTableTableName fields when joining two routes #8771
    • fix parsing of MySQL Server Version flag #8791
    • Fix typo in pool config syntax. Fixes #8819 #8820
    • VtGate schema tracker to use provided user #8857
    • fix merging dba query as a subquery both in v3 and gen4 #8871
    • sql_select_limit support #9027
    • [12.0] Handle sql_mode setting to compare modes in any order #9041
    • [12.0] Fix for schema tracker not recognizing RENAME for schema reloading #9043
    • [12.0] Fix savepoint support with reserved connections #9047
    • [12.0] Gen4: Handle single column vindex correctly with multi column #9061

    VReplication

    • MoveTables: don't create unnecessary streams on the target for non-intersecting sources and targets #8090
    • VDiff: Add BIT datatype to list of byte comparable types #8401
    • Fix vreplication error metric #8483
    • Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
    • Fix VReplication logging to file and db #8521
    • Fix passing the wrong cell/cells variable to CreateLookupVindex #8590
    • Refresh SrvVSchema after an ExternalizeVindex: was missing #8670
    • Fixing missing argument in function call #8692
    • Fix how we identify MySQL generated columns #8763

    VTAdmin

    • [vtadmin-web] Set Backup status indicators to the correct colours #8410
    • [vtadmin] Fix bad copy-paste in pool config flag parsing #8518

    vtctl

    • Print actual current datetime for vtctl dry run commands #8778

    vttestserver

    • Remove lingering mysqld sock files on vttestserver container startup #8463
    • Fixed filename for the configuration file in Vttestserver #8809

    CI/Build

    Build/CI

    • Update release process #7759
    • [ci] Add errcheck to golangci-lint #7996
    • gomod: do not replace GRPC #8416
    • consolidation: Fix flaky test #8417
    • Small build improvements #8418
    • proto: upgrade vtprotobuf version #8454
    • Automatically use the latest tag of Vitess for cluster upgrade E2E test #8471
    • Addition of known issues to release notes #8482
    • Fix Cluster 14 flakiness #8494
    • add question on backporting back to PR template #8541
    • Enable GitHub Actions concurrency feature #8546
    • Updated Makefile do_release script to include godoc steps #8550
    • Changing codahale/hdrhistogram import path #8637
    • Upgrade to Go 1.17 #8640
    • Fixes for reparent endtoend test flakiness #8642
    • hooks: remove govet because it already runs as part of golangci-lint #8674
    • Enhancement of the release instructions #8790
    • Remove consul-api usage in favor of official consul/api #8794
    • Enhancement of the release notes generation #8877

    Governance

    • update MAINTAINERS and CODEOWNERS for deepthi, pH14 and vmg #8675

    Query Serving

    • Cleanup: import of gh-ost test suite is complete #8459

    VReplication

    • VReplication support for non-PRIMARY KEY (Online DDL context) #8364

    Documentation

    Build/CI

    • maintainers: add Messaging as area of expertise #8578

    Cluster management

    • Enhance k8stopo flag documentation #8464
    • Update some vtctl/vtctlclient command help #8560
    • Trivial: fix the GenerateShardRanges vtctl[client] help #8586

    Examples

    • examples: update README to use new vreplication command syntax #8825

    Governance

    • update governance #8609

    Query Serving

    • vtexplain examples #8652

    VReplication

    • vtctl help/usage update #8879

    VTAdmin

    • [vtadmin] Remove outdated commands from vtadmin-web README #8496
    • Initial pass at some vtadmin docs #8526

    Enhancement

    Backup and Restore

    • Add Support for Restoring Specific Backups #8824

    Build/CI

    • Initial support for stress testing in end-to-end tests #8406
    • CI: Remove mariadb101 which is well past eol and no longer available in distributions #8446
    • Adding GitHub Self Hosted Runner tests #8721

    Cluster management

    • Vitess mixin improvements #7499
    • Expose topo_consul_lock_session_checks to allow customized consul session check #8298
    • Handle lock release with SIGHUP in VTGR #8472
    • Enhance PRS error message #8529
    • srvtopo: resilient watcher #8583
    • Add -restart_before_backup parameter for vtbackup #8608
    • srvtopo: allow unwatching from watch callbacks #8633
    • introduced cluster_operation as a new error code in vtrpc #8646
    • servenv: add --onclose_timeout flag #8651
    • Improve determinism of bootstrap #8840
    • OnlineDDL in v12: remove legacy code deprecated in earlier version #8972

    Observability

    • [vtadmin] cluster debug pages #8427
    • Export memstats for statsd backend #8777

    Query Serving

    • [tablet, queryrules] Extend query rules to check comments #8233
    • Add "show global status" support #8344
    • added batch lookup param to lookup vindex #8398
    • Online DDL/VReplication: able to read from replica #8405
    • multishard autocommit should work for bypass routing #8428
    • Add "no-scatter" flag to prohibit the use of scatter queries #8439
    • OnlineDDL: -skip-topo is always 'true' #8450
    • Add rows as keyword in sqlparser #8467
    • Periodic update of gh-ost binary #8470
    • fix when show tables record too much error log "Got unhandled packet.." #8478
    • SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8493
    • NativeDDL: analyzing added&removed unique keys #8495
    • Refactor authentication server plugin mechanism #8503
    • Update to planetscale/tengo v0.10.1-ps.v4 #8516
    • query serving to continue when topo server restarts - main #8534
    • support grpc reflection on APIs #8551
    • Customized CreateLookUpVindex to be Compatible with non-consistent Lookup Vindex #8570
    • Add additional options for configuring SSL modes as a client #8588
    • Fail plan for unsupported aggregate function #8593
    • gateway: use keyspace events when buffering requests #8700
    • Add optional query annotations, i.e. prefix SQL comments indicating the #8783
    • Slight improvement of the ACL error message #8805
    • Changed parsing for unions #8821
    • Use vt_dba user for online schema migration cutover phase #8836
    • Add parsing support for column list with subquery alias #8884
    • Add planner-version flag to vtexplain #8979
    • [Backport 12.0]: Add planner-version flag to vtexplain #9003

    TabletManager

    • Allow min TLS version for tablet to mysqld conns #8757

    VReplication

    • VReplication: Add ability to tag workflows #8388
    • Ignore SBR statements from pt-table-checksum #8396
    • Tablet Picker: add metric to record lack of available tablets #8403
    • VReplication: support different keys on source and target tables (different covered columns) #8423
    • VStream API: handle reparenting and unhealthy tablets #8445
    • Change local example to use v2 vreplication flows and change default to v2 from v1 #8553
    • Customized CreateLookUpVindex to Include a Flag to Continue Vreplication After Backfill with an Owner Provided #8572
    • Update MoveTables CLI help to reflect change we made at the beginning of #8597
    • VStreamer Field/Row Events: add Keyspace/Shard #8598
    • VStream API: Add flag to stop streaming on a reshard #8628

    VTAdmin

    • [vtadmin] cluster rpc pools #8421
    • Add support for passing custom interceptors to vtadmin grpcserver #8507

    Feature Request

    Cluster management

    • [vtctldserver] Migrate remaining ServingGraph rpcs to VtctldServer #8249
    • [grpctmclient] Add support for (bounded) per-host connection reuse #8368
    • VTGR: Vitess + MySQL group replication #8387
    • Add isActive flag to vtgr to support multi-cell topology #8780
    • [vtctld] Add SetWritable, StartReplication and StopReplication rpcs #8816
    • [vtctld] sleep/ping tablets #8826
    • [vtctld] migrate more util rpcs #8843
    • [vtctld] migrate validator rpcs #8849
    • [vtctld] localvtctldclient #8882
    • [vtctldserver] Migrate RunHealthCheck #8892
    • [vtctl] run new commands as standalone binary #8893

    VTAdmin

    • [vtadmin] [experimental] add per-api RBAC implementation #8515
    • [vtadmin] shard replication positions #8775
    • [vtadmin] GetVtctlds #8792

    Internal Cleanup

    Build/CI

    • Bump aws-sdk-go to v1.34.2 #8632
    • Upgrade consul api: go get github.com/hashicorp/consul/[email protected] #8784

    Cluster management

    • topodata: remove deprecated field 'ServedTypes' #8566
    • Upgrade to k8s 1.18 client #8762
    • [vtctl] command deprecations #8967

    Deployments

    • Remove the deprecated Helm charts and related code #8868

    Examples

    • Bump ini from 1.3.5 to 1.3.8 in /vitess-mixin/e2e #8823
    • backport operator updates to v12 branch . #9057

    General

    • Migrate k8s topo CRD to v1 api (v12 branch) #9046

    Governance

    • fix copyright #8611

    Query Serving

    • Adds flag to vttablet to disallow online DDL statements #8433
    • Remove vtrpc.LegacyErrorCode #8456
    • Allow for configuration of the minimal TLS version #8460
    • engine: allow retrying partial primitives #8727
    • srvtopo: expose WatchSrvKeyspace #8752
    • Clean up Primitive interface implementations #8901

    VReplication

    • LegacySplitCloneWorker is no longer used in any code paths #8867

    VTAdmin

    • [vtadmin] Add Options struct to wrap grpc/http options #8461
    • Update vtadmin local scripts to enable basic rbac #8801

    vtctl

    • Correctly identify backup timestamp var as a string #8891

    Other

    Examples

    • Improve the Docker local and compose examples #8685

    Performance

    Cluster management

    • throttler: don't allocate any resources unless it is actually enabled #8643

    Query Serving

    • cache: track memory usage more closely #8804

    VTAdmin

    • Use fmt.Fprintf instead of fmt.Sprintf #8922

    Testing

    Cluster management

    • Alter tests to reuse cluster components #8276
    • Adds some more orchestrator tests to vtorc #8535

    Query Serving

    • increase conn killer check to double the tx timeout value #8649
    • Added UNION testcases and auxilary code for running tests #8797
    • fixed regression in v3 for grouping by integer functions #8856

    VTAdmin

    • [vtadmin] Add a vtctld Dialer unit test #8455

    The release includes 1351 commits (excluding merges)

    Thanks to all our contributors: @Anders-PlanetScale, @GuptaManan100, @Johnny-Three, @Juneezee, @ajm188, @aquarapid, @askdba, @bnu0, @carsonoid, @choo-stripe, @dbussink, @dctrwatson, @deepthi, @demmer, @dependabot[bot], @derekperkins, @doeg, @eeSeeGee, @falun, @fatih, @frouioui, @hallaroo, @harshit-gangal, @ilikeorangutans, @mattlord, @rafael, @ritwizsinha, @rohit-nayak-ps, @sahutd, @shlomi-noach, @sonne5, @systay, @tdakkota, @tokikanno, @utk9, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.0-de06fca.tar.gz(400.35 MB)
    vitess-12.0.0-de06fca.x86_64.rpm(398.64 MB)
    vitess_12.0.0-de06fca_amd64.deb(400.36 MB)
  • v12.0.0-rc1(Oct 5, 2021)

    Release of Vitess v12.0.0

    Major Changes

    This release includes the following major changes or new features.

    Inclusive Naming

    A number of commands and RPCs have been deprecated as we move from master to primary. All functionality is backward compatible except as noted under Incompatible Changes.

    Gen4 Planner

    The newest version of the query planner, Gen4, becomes an experimental feature as part of this release. While Gen4 has been under development for a few release cycles, we have now reached parity with its predecessor, v3.

    To use Gen4, VTGate's -planner_version flag needs to be set to gen4.

    Minor Changes

    Query Serving

    • Add SHOW VITESS_REPLICATION_STATUS Query Support #8900
      • The new SHOW VITESS_REPLICATION_STATUS command/query shows the MySQL replica/replication (not vreplication) health for the vitess deployment.

    Incompatible Changes

    vtctl command output

    Wherever vtctl commands produced master or MASTER for tablet type, they now produce primary or PRIMARY. Scripts and tools that depend on parsing command output will need to change.

    Example:

    $ vtctlclient -server localhost:15999 ListAllTablets 
    zone1-0000000100 sourcekeyspace 0 primary 192.168.0.134:15100 192.168.0.134:17100 [] 2021-09-24T01:12:00Z
    zone1-0000000101 sourcekeyspace 0 rdonly 192.168.0.134:15101 192.168.0.134:17101 [] <null>
    zone1-0000000102 sourcekeyspace 0 replica 192.168.0.134:15102 192.168.0.134:17102 [] <null>
    zone1-0000000103 sourcekeyspace 0 rdonly 192.168.0.134:15103 192.168.0.134:17103 [] <null>
    

    Governance

    • Added @GuptaManan100 to maintainers #8833

    Changelog

    Bug fixes

    Build/CI

    • Update golang/x dependencies #8688
    • tests: use AtomicInt32 instead of int to fix races #8696

    Cluster management

    • Use consul lock properly #8310
    • Trivial: mysqlctl reinit_config -h was not showing help #8369
    • Port #8422 to main branch #8745
    • [vtctl] Add missing call to flag.Parse in commandGetRoutingRules #8795
    • Fix for padding in OrderAndCheckPartitions #8873

    Observability

    • schema engine: reset table size stats when a table is dropped #8634

    Query Serving

    • Fix for function calls in DEFAULT value of CREATE TABLE statement in main #8477
    • Fixing multiple issues related to onlineddl/lifecycle #8500
    • boolean values should not be parenthesised in default clause #8502
    • mysql/server: abort connection if secure transport is required #8504
    • Clear only the current shard's schemacopy rows during schema reload #8519
    • go/mysql/auth: fix error message for wrong auth method. #8525
    • Fix handshake protocol with MySQL client #8537
    • Copy parameter data from byte buffer in COM_STMT_SEND_LONG_DATA #8562
    • Negative float default #8587
    • [vtexplain] prevent panic on show tables/show full tables #8601
    • OnlineDDL: better scheduling/cancellation logic #8603
    • Fix parsing of FIRST clause in ALTER TABLE #8614
    • onlineddl Executor: build schema with DBA user #8624
    • Handle subquery merging with references correctly #8662
    • Fixing error on deletes from owning table for a lookup being populated by CreateLookupVindex after a SwitchWrites #8701
    • Fixing a panic in vtgate with OLAP mode #8722
    • default to primary tablet if not set in VStream api #8755
    • Fix bug in parsing of version comments #8770
    • Proper merge of the SysTableTableName fields when joining two routes #8771
    • fix parsing of MySQL Server Version flag #8791
    • Fix typo in pool config syntax. Fixes #8819 #8820
    • VtGate schema tracker to use provided user #8857
    • fix merging dba query as a subquery both in v3 and gen4 #8871

    VReplication

    • MoveTables: don't create unnecessary streams on the target for non-intersecting sources and targets #8090
    • VDiff: Add BIT datatype to list of byte comparable types #8401
    • Fix vreplication error metric #8483
    • Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
    • Fix VReplication logging to file and db #8521
    • Fix passing the wrong cell/cells variable to CreateLookupVindex #8590
    • Refresh SrvVSchema after an ExternalizeVindex: was missing #8670
    • Fixing missing argument in function call #8692
    • Fix how we identify MySQL generated columns #8763

    VTAdmin

    • [vtadmin-web] Set Backup status indicators to the correct colours #8410
    • [vtadmin] Fix bad copy-paste in pool config flag parsing #8518

    vtctl

    • Print actual current datetime for vtctl dry run commands #8778

    vttestserver

    • Remove lingering mysqld sock files on vttestserver container startup #8463
    • Fixed filename for the configuration file in Vttestserver #8809

    CI/Build

    Build/CI

    • Update release process #7759
    • [ci] Add errcheck to golangci-lint #7996
    • gomod: do not replace GRPC #8416
    • consolidation: Fix flaky test #8417
    • Small build improvements #8418
    • proto: upgrade vtprotobuf version #8454
    • Automatically use the latest tag of Vitess for cluster upgrade E2E test #8471
    • Addition of known issues to release notes #8482
    • Fix Cluster 14 flakiness #8494
    • add question on backporting back to PR template #8541
    • Enable GitHub Actions concurrency feature #8546
    • Updated Makefile do_release script to include godoc steps #8550
    • Changing codahale/hdrhistogram import path #8637
    • Upgrade to Go 1.17 #8640
    • Fixes for reparent endtoend test flakiness #8642
    • hooks: remove govet because it already runs as part of golangci-lint #8674
    • Enhancement of the release instructions #8790
    • Remove consul-api usage in favor of official consul/api #8794
    • Enhancement of the release notes generation #8877

    Governance

    • update MAINTAINERS and CODEOWNERS for deepthi, pH14 and vmg #8675

    Query Serving

    • Cleanup: import of gh-ost test suite is complete #8459

    VReplication

    • VReplication support for non-PRIMARY KEY (Online DDL context) #8364

    Documentation

    Build/CI

    • maintainers: add Messaging as area of expertise #8578

    Cluster management

    • Enhance k8stopo flag documentation #8464
    • Update some vtctl/vtctlclient command help #8560
    • Trivial: fix the GenerateShardRanges vtctl[client] help #8586

    Examples

    • examples: update README to use new vreplication command syntax #8825

    Governance

    • update governance #8609

    Query Serving

    • vtexplain examples #8652

    VReplication

    • vtctl help/usage update #8879

    VTAdmin

    • [vtadmin] Remove outdated commands from vtadmin-web README #8496
    • Initial pass at some vtadmin docs #8526

    Enhancement

    Backup and Restore

    • Add Support for Restoring Specific Backups #8824

    Build/CI

    • Initial support for stress testing in end-to-end tests #8406
    • CI: Remove mariadb101 which is well past eol and no longer available in distributions #8446
    • Adding GitHub Self Hosted Runner tests #8721

    Cluster management

    • Vitess mixin improvements #7499
    • Expose topo_consul_lock_session_checks to allow customized consul session check #8298
    • Handle lock release with SIGHUP in VTGR #8472
    • Enhance PRS error message #8529
    • srvtopo: resilient watcher #8583
    • Add -restart_before_backup parameter for vtbackup #8608
    • srvtopo: allow unwatching from watch callbacks #8633
    • introduced cluster_operation as a new error code in vtrpc #8646
    • servenv: add --onclose_timeout flag #8651
    • Improve determinism of bootstrap #8840

    Observability

    • [vtadmin] cluster debug pages #8427
    • Export memstats for statsd backend #8777

    Query Serving

    • [tablet, queryrules] Extend query rules to check comments #8233
    • Add "show global status" support #8344
    • added batch lookup param to lookup vindex #8398
    • Online DDL/VReplication: able to read from replica #8405
    • multishard autocommit should work for bypass routing #8428
    • Add "no-scatter" flag to prohibit the use of scatter queries #8439
    • OnlineDDL: -skip-topo is always 'true' #8450
    • Add rows as keyword in sqlparser #8467
    • Periodic update of gh-ost binary #8470
    • fix when show tables record too much error log "Got unhandled packet.." #8478
    • SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8493
    • NativeDDL: analyzing added&removed unique keys #8495
    • Refactor authentication server plugin mechanism #8503
    • Update to planetscale/tengo v0.10.1-ps.v4 #8516
    • query serving to continue when topo server restarts - main #8534
    • support grpc reflection on APIs #8551
    • Customized CreateLookUpVindex to be Compatible with non-consistent Lookup Vindex #8570
    • Add additional options for configuring SSL modes as a client #8588
    • Fail plan for unsupported aggregate function #8593
    • gateway: use keyspace events when buffering requests #8700
    • Add optional query annotations, i.e. prefix SQL comments indicating the #8783
    • Slight improvement of the ACL error message #8805
    • Changed parsing for unions #8821
    • Use vt_dba user for online schema migration cutover phase #8836
    • Add parsing support for column list with subquery alias #8884

    TabletManager

    • Allow min TLS version for tablet to mysqld conns #8757

    VReplication

    • VReplication: Add ability to tag workflows #8388
    • Ignore SBR statements from pt-table-checksum #8396
    • Tablet Picker: add metric to record lack of available tablets #8403
    • VReplication: support different keys on source and target tables (different covered columns) #8423
    • VStream API: handle reparenting and unhealthy tablets #8445
    • Change local example to use v2 vreplication flows and change default to v2 from v1 #8553
    • Customized CreateLookUpVindex to Include a Flag to Continue Vreplication After Backfill with an Owner Provided #8572
    • Update MoveTables CLI help to reflect change we made at the beginning of #8597
    • VStreamer Field/Row Events: add Keyspace/Shard #8598
    • VStream API: Add flag to stop streaming on a reshard #8628

    VTAdmin

    • [vtadmin] cluster rpc pools #8421
    • Add support for passing custom interceptors to vtadmin grpcserver #8507

    Feature Request

    Cluster management

    • [vtctldserver] Migrate remaining ServingGraph rpcs to VtctldServer #8249
    • [grpctmclient] Add support for (bounded) per-host connection reuse #8368
    • VTGR: Vitess + MySQL group replication #8387
    • Add isActive flag to vtgr to support multi-cell topology #8780
    • [vtctld] Add SetWritable, StartReplication and StopReplication rpcs #8816
    • [vtctld] sleep/ping tablets #8826
    • [vtctld] migrate more util rpcs #8843
    • [vtctld] migrate validator rpcs #8849
    • [vtctld] localvtctldclient #8882
    • [vtctldserver] Migrate RunHealthCheck #8892
    • [vtctl] run new commands as standalone binary #8893

    VTAdmin

    • [vtadmin] [experimental] add per-api RBAC implementation #8515
    • [vtadmin] shard replication positions #8775
    • [vtadmin] GetVtctlds #8792

    Internal Cleanup

    Build/CI

    • Bump aws-sdk-go to v1.34.2 #8632
    • Upgrade consul api: go get github.com/hashicorp/consul/[email protected] #8784

    Cluster management

    • topodata: remove deprecated field 'ServedTypes' #8566
    • Upgrade to k8s 1.18 client #8762

    Deployments

    • Remove the deprecated Helm charts and related code #8868

    Examples

    • Bump ini from 1.3.5 to 1.3.8 in /vitess-mixin/e2e #8823

    Governance

    • fix copyright #8611

    Query Serving

    • Adds flag to vttablet to disallow online DDL statements #8433
    • Remove vtrpc.LegacyErrorCode #8456
    • Allow for configuration of the minimal TLS version #8460
    • engine: allow retrying partial primitives #8727
    • srvtopo: expose WatchSrvKeyspace #8752
    • Clean up Primitive interface implementations #8901

    VReplication

    • LegacySplitCloneWorker is no longer used in any code paths #8867

    VTAdmin

    • [vtadmin] Add Options struct to wrap grpc/http options #8461
    • Update vtadmin local scripts to enable basic rbac #8801

    vtctl

    • Correctly identify backup timestamp var as a string #8891

    Other

    Examples

    • Improve the Docker local and compose examples #8685

    Performance

    Cluster management

    • throttler: don't allocate any resources unless it is actually enabled #8643

    Query Serving

    • cache: track memory usage more closely #8804

    VTAdmin

    • Use fmt.Fprintf instead of fmt.Sprintf #8922

    Testing

    Cluster management

    • Alter tests to reuse cluster components #8276
    • Adds some more orchestrator tests to vtorc #8535

    Query Serving

    • increase conn killer check to double the tx timeout value #8649
    • Added UNION testcases and auxilary code for running tests #8797
    • fixed regression in v3 for grouping by integer functions #8856

    VTAdmin

    • [vtadmin] Add a vtctld Dialer unit test #8455

    The release includes 1281 commits (excluding merges)

    Thanks to all our contributors: @Anders-PlanetScale, @GuptaManan100, @Johnny-Three, @Juneezee, @ajm188, @aquarapid, @askdba, @bnu0, @choo-stripe, @dbussink, @dctrwatson, @deepthi, @demmer, @dependabot[bot], @derekperkins, @doeg, @eeSeeGee, @falun, @fatih, @frouioui, @hallaroo, @harshit-gangal, @ilikeorangutans, @mattlord, @rafael, @ritwizsinha, @rohit-nayak-ps, @sahutd, @shlomi-noach, @sonne5, @systay, @tdakkota, @tokikanno, @utk9, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.0-rc1-bd0a4ed.tar.gz(401.79 MB)
    vitess-12.0.0_rc1-bd0a4ed.x86_64.rpm(400.06 MB)
    vitess_12.0.0-rc1-bd0a4ed_amd64.deb(401.79 MB)
  • v11.0.1(Sep 9, 2021)

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v11.0.1, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v11.0.4 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Bug fixes

    Cluster management

    • Fix problematic ERS cases #8744

    Query Serving

    • Handle subquery merging with references correctly #8661
    • onlineddl Executor: build schema with DBA user #8667
    • Backport to 11: Fixing a panic in vtgate with OLAP mode #8746
    • Backport into 11: default to primary tablet if not set in VStream api #8766

    VReplication

    • Refresh SrvVSchema after an ExternalizeVindex: was missing #8669

    CI/Build

    Build/CI

    • Vitess Release 11.0.0 #8549
    • Backport to 11: Updated Makefile do_release script to include godoc steps #8787

    The release includes 18 commits (excluding merges) Thanks to all our contributors: @aquarapid, @askdba, @frouioui, @harshit-gangal, @rohit-nayak-ps, @shlomi-noach, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.1-92ac1ff.tar.gz(397.08 MB)
    vitess-11.0.1-92ac1ff.x86_64.rpm(394.80 MB)
    vitess_11.0.1-92ac1ff_amd64.deb(397.09 MB)
  • v11.0.0(Jul 27, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Known Issues

    • A critical vulnerability CVE-2021-44228 in the Apache Log4j logging library was disclosed on Dec 9 2021. The project provided release 2.15.0 with a patch that mitigates the impact of this CVE. It was quickly found that the initial patch was insufficient, and additional CVEs CVE-2021-45046 and CVE-2021-44832 followed. These have been fixed in release 2.17.1. This release of Vitess, v11.0.0, uses a version of Log4j below 2.17.1, for this reason, we encourage you to use version v11.0.4 instead, to benefit from the vulnerability patches.

    • An issue where the value of the -force flag is used instead of -keep_data flag's value in v2 vreplication workflows (#9174) is known to be present in this release. A workaround is available in the description of issue #9174.

    Bug fixes

    Build/CI

    • update moby/term to fix darwin build issue #7787
    • Removing Linux/amd64 specific subdependencies #7796
    • CI: run some tests inside docker to workaround GH Actions issue #7868
    • Fix flaky race condition in vtexplain #7930
    • Fixing flaky endtoend/tablegc tests #7947
    • Add libwww-perl to allow pt-osc wrapper to work in vitess/lite #8141

    Cluster management

    • Add missing return on a failed GetShard call in FindAllShardsInKeyspace #7992
    • Increase the default srv_topo_timeout #8011
    • parser: support index name in FOREIGN KEY clause; Online DDL to reject FK clauses #8058
    • Correctly parse cell names with dashes in tablet aliases #8167
    • /healthz should not report ok when vttablet is not connected to mysql #8238
    • [topo] Refactor ExpandCells to not error on valid aliases #8291
    • tm state: don't populate metadata in updateLocked #8362
    • [grpcvtctldserver] Fix backup detail limit math #8402

    Query Serving

    • fix select star, col1 orderby col1 bug. #7743
    • VTExplain: Add support for multi table join query #7829
    • Routing of Information Schema Queries #7841
    • Fix + integration test for keyspaces_to_watch routing regression [fixes #7882] #7873
    • Revert "[tablet, queryrules] Extend query rules to check comments" #7897
    • vtctl: return error on invalid ddl_strategy #7923
    • Panic when EOF after @ symbol #7925
    • Fixes encoding of sql strings #8033
    • Make TwoPC engine initialization async. #8048
    • Fix for issue with information_schema queries with both table name and schema name predicates #8087
    • Fix for transactions not allowed to finish during PlannedReparentShard #8089
    • Set fully parsed to false when ignoring errors #8094
    • Fix for query and sub query with limits #8097
    • Fix buffering when using reserved connections #8102
    • Parse generated columns in DDL #8117
    • More explicit message on online DDL parsing error #8118
    • healthcheck: attempt to update primary only if the current tablet is serving #8121
    • PRIMARY in index hint list for master #8160
    • Signed int parse #8189
    • Delete table reference alias support #8393
    • Fix for function calls in DEFAULT value of CREATE TABLE statement in release-11.0 #8476
    • Backport: Fixing multiple issues related to onlineddl/lifecycle #8517
    • boolean values should not be parenthesised in default clause - release 11 #8531

    VReplication

    • VDiff: Use byte compare if weight_string() returns null for either source or target #7696
    • Rowlog: Update rowlog for the API change made for the vstream skew alignment feature #7809
    • Pad binlog values for binary() columns to match the value returned by mysql selects #7969
    • Change vreplication error metric name to start with a string to pass prometheus validations #7983
    • Pass the provided keyspace through to UpdateDisableQueryService rather than hard-coding the sourceKeyspace #8020
    • Fix vreplication timing metrics #8024
    • Switchwrites: error if no tablets available on target for reverse replication #8142
    • Remove noisy vexec logs #8144
    • VReplicationExec: don't create new stats objects on a select #8166
    • Binlog JSON Parser: handle inline types correctly for large documents #8187
    • Schema Tracking Flaky Test: Ignore unrelated gtid progress events #8283
    • Adds padding to keyrange comparison #8296
    • VReplication Reverse Workflows: add keyspace scope to vindex while creating reverse vreplication streams #8385
    • OnlineDDL/Vreplication stress test: investigating failures #8390
    • Ignore SBR statements from pt-table-checksum #8396
    • Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
    • VDiff: Add BIT datatype to list of byte comparable types #8401
    • Fix excessive VReplication logging to file and db #8521

    VTAdmin

    • Add missing return in vtctld-* DSN case, and log any flag that gets ignored #7872
    • [vtadmin-web] Do not parse numbers/booleans in URL query parameters by default #8100
    • [vtadmin-web] Small bugfix where stream source displayed instead of target #8311

    CI/Build

    Build/CI

    • Planbuilder: Fix fuzzer #7952
    • java: Bump SNAPSHOT version to 11.0.0-SNAPSHOT after Vitess release v10 #7968
    • Add optional TLS feature to gRPC servers #8049
    • Add image and base image arguments to build.sh #8064
    • [fuzzing] Add report #8128
    • CI: fail PR if it does not have required labels #8147
    • trigger pr-labels workflow when labels added/removed from PR #8157
    • Add vmg as a maintainer #8186
    • Refactor vtorc endtoend tests #8215
    • Use shared testing helper which guards against races in tmclient construction #8300
    • moved multiple vtgate and tabletgateway to individual shards #8305
    • include squashed PRs in release notes #8326
    • pr-labels workflow: only check actual PRs #8327
    • Heuristic fix for a flaky test: allow for some noise to pass through #8339
    • ci: upgrade to Go 1.16 #8274
    • mod: upgrade etcd to stable version #8357
    • Resolve go.mod issue with direct dependency on planetscale/tengo #8383
    • Initial GitHub Docker Build Setup #8399

    Cluster management

    • make: build vitess as static binaries by default #7795
    • Fix TableGC flaky test by reducing check interval #8270

    Java

    • Bump commons-io from 2.6 to 2.7 in /java #7953

    Other

    • Bump MySQL version for docker builds to 8.0.23 #7811

    Query Serving

    • Online DDL Vreplication test suite: adding tests #8213
    • Online DDL/VReplication: more passing tests (no UK/PK) #8225
    • Online DDL: reject ALTER TABLE...RENAME statements #8227
    • Online DDL/VReplication: fail on existence of FOREIGN KEY #8228
    • Online DDL/VReplication: test PK column case change #8336
    • Online DDL/VReplication: add PK DATETIME->TIMESTAMP test #8338
    • Bugfix: assign cexpr.references for column CONVERTed to utf8mb4 #8355

    VReplication

    • Towards a VReplication/OnlineDDL testing suite #8181
    • Online DDL/Vreplication: column type awareness #8239
    • remove duplicate ReadMigrations, fixing build #8258
    • VReplication (and by product, Online DDL): support GENERATED column as part of PRIMARY KEY #8335

    vttestserver

    • docker/vttestserver: Set max_connections in default-fast.cnf at container build time #7810

    Documentation

    Cluster management

    • Enhance k8stopo flag documentation #8458

    Build/CI

    • Update version for latest snapshot #7801
    • v10 GA Release Notes #7964
    • Updates and Corrections to v9 Release Notes #8295

    Query Serving

    • Update link to 'Why FK not supported in Online DDL' blog post #8372

    Other

    • correct several wrong words #7822
    • MAINTAINERS.md: Update enisoc's email. #7909
    • Modification of the Markdown list format in the release notes template #7962
    • Update vreplicator docs #8014
    • Release notes: add Known Issues #8027

    Enhancement

    Observability

    • [trace] Add logging support to opentracing plugins #8289

    Build/CI

    • Makefile: add cross-build target for cross-compiling client binaries #7806
    • git: improve signoff detection #7852
    • Release notes generation #7932
    • vitess/lite docker build fails for mysql80 #7943
    • Fix a gofmt warning #7959
    • docker/vttestserver/run.sh: Add $CHARSET environment variable #7970
    • docker/lite/install_dependencies.sh: If the dependency install loop reaches its max number of retries, consider that a failure; exit the script nonzero so the build halts. #7976
    • Add commit count and authors to the release notes #7982
    • Automate version naming and release tagging #8034
    • Make sure to only allow codegen when the rest of the code compiles #8169
    • Add options to vttestserver to pass -foreign_key_mode, -enable_online_ddl, and -enable_direct_ddl through to vtcombo #8177

    Cluster management

    • Dynamic throttle metric threshold #7742
    • Bump Bootstrap version, per CVE-2018-14040 (orchestrator/vtorc) #7824
    • Check tablet alias before removing after error stream #7915
    • vttablet/tabletmanager - add additional test for tmstate.Open() #7993
    • Add vttablet_restore_done hook #8007
    • Added ValidateVSchema #8012
    • Add RemoteOperationTimeout to both legacy and grpc ChangeTabletType implementations. #8052
    • [vtctldserver] Add guard against self-reparent, plus misc updates #8084
    • [tm_state] updateLocked should re-populate local metadata tables to reflect promotion rule changes #8107
    • [vtctld] Migrate ApplyVSchema to VtctldServer #8113
    • vtctl.generateShardRanges -> key.GenerateShardRanges #8134
    • etcd: add grpc.WithBlock to client config #8205
    • [workflow] Add tracing to GetWorkflows endpoint #8266
    • [vtctldclient] Add legacy shim #8284
    • [vtctldserver] Add tracing #8285
    • [vtctldserver] Add additional backup info fields #8321
    • [workflow] Call scanWorkflow concurrently #8272

    Query Serving

    • [Gen4] Implemented table alias functionality in the semantic analysis #7629
    • Improved error messages (tabletserver) #7747
    • Allow modification of tablet unhealthy_threshold via debugEnv #7753
    • [Gen4] Initial Horizon Planning #7775
    • [tablet, queryrules] Extend query rules to check comments #7784
    • Add support for showing global gtid executed per shard #7856
    • Minor cleanups around errors on the vtgate #7864
    • log unsupported queries #7865
    • Show databases like #7912
    • Add rank as reserved keyword #7944
    • Online DDL: introducing ddl_strategy -singleton-context flag #7946
    • Ignore the error and log as warn if not able to validate the current system setting for check and ignore case #8004
    • Detect and signal schema changes on vttablets #8005
    • DDL bypass plan #8013
    • Online DDL: progress & ETA for Vreplication migrations #8015
    • Scatter errors as warning in olap query #8018
    • livequeryz: livequeryz/terminate link should be relative #8025
    • Handle online DDL user creation if we do not have SUPER privs (e.g. AWS Aurora) #8038
    • Update the schema copy with minimal changes #8067
    • Gen4: Support for order by column not present in projection #8070
    • Schema tracking in vtgate #8074
    • protobuf: upgrade #8075
    • added mysql 8.0 reserved keywords #8086
    • Fix ghost/pt-osc in the external DB case where MySQL might be reporting #8115
    • Support for vtgate -enable_direct_ddl flag #8116
    • add transaction ID to query log for acquisition and analysis #8133
    • Inline reference #8136
    • Improve ScatterErrorsAsWarnings functionality #8139
    • Gen4: planning Select IN #8155
    • Primary key name #8188
    • Online DDL: read and publish gh-ost FATAL message where possible #8192
    • Expose inner net.Conn to be able to write better unit tests #8217
    • vtgate: validate important flags during startup #8218
    • Online DDL/VReplication: AUTO_INCREMENT support and tests #8223
    • [Gen4] some renaming from v4 to Gen4 #8234
    • Schema Tracking: Optimize and Bug Fix #8243
    • gen4: minor refactoring to improve readability #8245
    • gen4: plan more opcodes #8254
    • Online DDL: report rows_copied for migration (initial support in gh-ost) #8255
    • Schema tracking: new tables in sharded keyspace #8256
    • [parser] use table_alias for ENGINE option in CREATE TABLE stmt #8307
    • Add /debug/env for vtgate #8292
    • gen4: outer joins #8312
    • Gen4: expand star in projection list #8325
    • gen4: Fail all queries not handled well by gen4 #8359
    • Gen4 fail more2 #8382
    • SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8532
    • [11.0] query serving to continue when topo server restarts #8533
    • [11.0] Disable allowing set statements on system settings by default #8540

    VReplication

    • Change local example to use v2 vreplication flows and make v2 flows as the default #8527
    • Use Dba user when Vexec is runAsAdmin #7731
    • Add table for logging stream errors, state changes and key workflow steps #7831
    • Fix some static check warning #7960
    • VSchema Validation on ReshardWorkflow Creation #7977
    • Tracking rows_copied #7980
    • Vdiff formatting improvements #8079
    • Ignore generated columns in workflows #8129
    • VReplication Copy Phase: Increase default replica lag tolerance. Also make it and copy timeout modifiable via flags #8130
    • Materialize: Add additional comparison operators in Materialize and fix bug where they not applied for sharded keyspaces #8247
    • Copy Phase: turn on OptimizeInserts by default #8248
    • Tracker/VStreamer: only reload schema for tables in current database and not for internal table artifacts #8257
    • Online DDL/Vreplication suite: support ENUM->VARCHAR/TEXT type change #8275
    • Added TableName to DiffReport struct. #8279
    • New VReplication lag metric #8306
    • VStream API: add heartbeat for idle streams #8244
    • Online DDL/VReplication: support non-UTF8 character sets #8322
    • Online DDL/Vreplication suite: fix test for no shared UK #8334
    • Online DDL/VReplication: support DROP+ADD column of same name #8337
    • Online DDL/VReplication test suite: support ENUM as part of PRIMARY KEY #8345
    • Change local example to use v2 vreplication flows #8527
    • Tablet Picker: add metric to record lack of available tablets #8403

    VTAdmin

    • [vtadmin-web] Add useSyncedURLParam hook to persist filter parameter in the URL #7857
    • [vtadmin-web] Display more data on /gates view and add filtering #7876
    • [vtadmin] Promote ErrNoSchema to a TypedError which returns http 404 #7885
    • [vtadmin] gate/tablet/vtctld FQDNs #7886
    • [vtadmin-web] Display vindex data on Schema view #7917
    • [vtadmin-web] Add filtering and source/target shards to Workflows view #7948
    • [vtadmin-web] Add filtering + shard counts/status to Keyspaces view #7991
    • [vtadmin-web] Display shard state on Tablets view + extract tablet utilities #7999
    • [vtadmin] experimental tabletdebug #8003
    • [vtadmin-web] Display timestamps + stream counts on Workflows view #8009
    • [vtadmin-web] Updates to table styling + other CSS #8072
    • [vtadmin-web] Add Tab components #8119
    • [vtadmin-api] Update GetTablet to use alias instead of hostname #8163
    • [vtadmin-api] Rename flag 'http-tablet-fqdn-tmpl' to 'http-tablet-url-tmpl' + update vtadmin flags for local example #8164
    • Add --tracer flag to vtadmin and actually start tracing #8165
    • [vtadmin-web] The hastiest Tablet view (+ experimental debug vars) #8170
    • [vtadmin-web] Add tabs to Workflow view #8203
    • [vtctld] Add GetSrvVSchemas command #8221
    • [vtadmin-api] Add HTTP endpoints for /api/srvvschemas and /api/srvvschema/{cluster}/{cell} #8226
    • [vtadmin-web] Add QPS and VReplicationQPS charts to Tablet view #8263
    • [vtadmin-web] Add client-side error handling interface + Bugsnag implementation #8287
    • [vtadmin-web] Add chart for stream vreplication lag across all streams in a workflow #8331
    • [vtctldproxy] Add more annotations to vtctld Dial calls #8346

    vttestserver

    • Vttest create db #7989
    • Adds an environment variable to set the MySQL max connections limit in vttestserver docker image #8210

    Feature Request

    Build/CI

    • [trace] Add optional flags to support flexible jaeger configs #8199

    Cluster management

    • Add VtctldServer to vtcombo #7896
    • [vtctldserver] Migrate routing rules RPCs, and also RebuildVSchemaGraph #8197
    • [vtctldserver] Migrate CellInfo, CellAlias rw RPCs #8219
    • [vtctldserver] Add RefreshState RPCs #8232

    Query Serving

    • VTGate grpc implementation of Prepare and CloseSession #8211
    • Schema tracking: One schema load at a time per keyspace #8224

    VTAdmin

    • [vtadmin-web] Add DataFilter + Workspace layout components #8032
    • [vtadmin-web] Add Tooltip + HelpTooltip components #8076
    • [vtadmin-web] Add initial Stream view, render streams on Workflow view #8091
    • [vtadmin-web] The hastiest-ever VTExplain UI #8092
    • [vtadmin-web] Add source-map-explorer util #8093
    • [vtadmin-web] Add Keyspace detail view #8111
    • [vtadmin-api] Add GetKeyspace endpoint #8125
    • [workflow] Add vreplication_log data to workflow protos, and VtctldServer.GetWorkflows method #8261
    • [vtadmin] Add debug endpoints #8268

    Internal Cleanup

    Build/CI

    • Makefile: fix cross-build comments #8246
    • remove unused hooks with refs to master branch #8250
    • [vtctldserver] Update tests to prevent races during tmclient init #8320
    • sks-keyservers.net seems to be finally dead, replace with #8363
    • endtoend: change log level of etcd start from error to info #8370

    Cluster management

    • Online DDL: code cleanup #7589
    • [wrangler|topotools] Migrate UpdateShardRecords, RefreshTabletsByShard, and {Get,Save}RoutingRules #7965
    • Tear down old stream_migrater shim, now that we're fully in package workflow #8073
    • [mysqlctl] Restructure MetadataManager to reduce public API surface area #8152
    • naming: master to primary #8251
    • vtorc: code cleanup #8269

    Query Serving

    • Plan StreamExecute Queries #7941

    VReplication

    • [workflow] extract migration targets from wrangler #7934
    • [wrangler|workflow] Extract vrStream type to workflow.VReplicationStream #7966
    • [wrangler|workflow] Extract workflowState and workflowType out to package workflow #7967
    • [wrangler|workflow] extract *wrangler.streamMigrater to workflow.StreamMigrator #8008
    • [workflow] Migrate getCellsWith{Shard,Table}ReadsSwitched, TrafficSwitchDirection and TableRemovalType to package workflow #8190
    • [workflow] Cleanup wrangler wrappers, migrate checkIfJournalExistsOnTablet to package workflow #8193
    • Backports of #8403 #8483 #8489 #8401 #8521 #8396 from main into release 11.0 #8536

    VTAdmin

    • [vtadmin-api] Replace magic numbers with net/http constants #8127
    • [vtadmin-web] Move single-entity view components into subfolders #8202
    • [vtadmin] Ensure we log any errors when closing the tracer #8262

    Other

    Build/CI

    • add vtctldclient to binary directory #7889

    Cluster management

    • vttablet/tabletmanager: add isInSRVKeyspace/isShardServing #7929

    Other

    • Change VitessInputFormat key type #199
    • Online DDL plan via Send; "singleton" migrations on tablets #7785
    • flaky onlineddl tests: reduce -online_ddl_check_interval #7847
    • Looking into flaky endtoend upgrade test #7900
    • fixing flaky upgrade test #7901

    Query Serving

    • Introduce Concatenated Fixed-width Composite or CFC vindex #7537
    • Add common tags for stats backends that support it #7651
    • Add support for showing global vgtid executed #7797
    • perf: vttablet/mysql optimizations #7800
    • Fix bug with reserved connections to stale tablets #7879
    • Memory Sort to close the goroutines when callback returns error #7903
    • OnlineDDL: more migration check ticks upon migration start #7961
    • Update gh-ost binary to v1.1.3 #8021
    • VReplication Online DDL: fix classification of virtual columns #8043

    VReplication

    • VReplicationErrors metric: use . as delimiter instead of _ to behave well with Prometheus #7807

    VTAdmin

    • Update GetSchema filtering to exclude shards where IsMasterServing but no MasterAlias #7805
    • [vtadmin-api] Reintroduce include_non_serving_shards opt to GetSchema #7814
    • [vtadmin-web] Add DataCell component #7817
    • Rewrite useTableDefinitions hook as getTableDefinitions util #7821
    • [vtadmin-web] Display (approximate) table sizes + row count on /schemas view #7826
    • [vtadmin-web] Add Pip + TabletServingPip components #7827

    Performance

    Query Serving

    • vttablet: stream consolidation #7752
    • perf: optimize bind var generation #7828

    VReplication

    • Optimize the catchup phases by filtering out rows which not within range of copied pks during inserts #7708
    • Ability to compress gtid when stored in _vt.vreplication's pos column #7877
    • Performance & benchmarks (table copying) #7881
    • Dynamic packet sizing #7933
    • perf: vreplication client CPU usage #7951
    • VDIff: fix performance regression introduced by progress logging #8016
    • proto: Generate faster code using vtprotobuf #8173
    • proto: enable pooling for vreplication #8273

    Testing

    Build/CI

    • Attempt to fix TLS Server Flaky test #7842
    • TestMakeCommonTags Flaky Test: match elements #7843
    • upgrade tests: test against v9.0.0 #7848
    • FOSSA scan added #7862
    • ci: fix all racy tests #7904
    • CI: Revert docker change for unit tests from #7868 #7940
    • Add online ddl on start queries to schema list in vtexplain tablet #8397

    Java

    • [Java] JDBC mysql driver test #8154

    Other

    • Fuzzing: Fixup oss-fuzz build script #7782
    • Wrangler tests: Return a fake tablet in the wrangler test dialer to avoid tablet picker errors spamming the test logs #7863

    Query Serving

    • clean up test #7816
    • tabletserver: fix flaky test #7851
    • Planbuilder: Add fuzzer #7902
    • mysql: Small adjustments to fuzzer #7907
    • vtgate/engine: Add fuzzer #7914
    • Adding Fuzzer Test Cases #8106
    • Addition of fuzzer issues #8195

    VReplication

    • vstreamer: Add fuzzer #7918

    vttestserver

    • Speedup new vttestserver tests #8229
    • Vttestserver docker test #8253

    Cluster management

    • Make timestamp authoritative for master information #8381

    The release includes 1080 commits (excluding merges)

    Thanks to all our contributors: @AdamKorcz, @GuptaManan100, @Hellcatlk, @Johnny-Three, @acharisshopify, @ajm188, @alexrs, @aquarapid, @askdba, @deepthi, @doeg, @dyv, @enisoc, @frouioui, @gedgar, @guidoiaquinti, @harshit-gangal, @hkdsun, @idvoretskyi, @jmoldow, @kirs, @mcronce, @narcsfz, @noxiouz, @rafael, @rohit-nayak-ps, @setassociative, @shlomi-noach, @systay, @tokikanno, @vmg, @wangmeng99, @yangxuanjia, @zhangshj-inspur

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.0-aa798b8.tar.gz(397.07 MB)
    vitess-11.0.0-aa798b8.x86_64.rpm(394.82 MB)
    vitess_11.0.0-aa798b8_amd64.deb(397.08 MB)
simply SQL Parser for Go ( powered by vitess and TiDB )

vitess-sqlparser Simply SQL and DDL parser for Go (powered by vitess and TiDB ) this library inspired by https://github.com/xwb1989/sqlparser (origina

BlasTrain Co., Ltd. 303 Dec 5, 2022
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置,参数n为秒数,查询结

null 14 Nov 15, 2022
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

PingCAP 267 Nov 9, 2022
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice.

DoltHub 935 Dec 8, 2022
Interactive terminal user interface and CLI for database connections. MySQL, PostgreSQL. More to come.

?? dbui dbui is the terminal user interface and CLI for database connections. It provides features like, Connect to multiple data sources and instance

Kanan Rahimov 116 Dec 3, 2022
Golang restAPI crud project with mySql database.

Golang RestAPI using gorilla/mux Golang restAPI crud project with mySql database. Test Api with Thunder Client vs code beautiful Extension. and use Be

Md Abu. Raihan 6 Mar 26, 2022
A Go rest API project that is following solid and common principles and is connected to local MySQL database.

This is an intermediate-level go project that running with a project structure optimized RESTful API service in Go. API's of that project is designed based on solid and common principles and connected to the local MySQL database.

Kıvanç Aydoğmuş 21 Jun 6, 2022
CRUD API example is written in Go using net/http package and MySQL database.

GoCrudBook CRUD API example is written in Go using net/http package and MySQL database. Requirements Go MySQL Code Editor Project Structure GoCrudBook

Serhat Karabulut 4 Aug 1, 2022
A proxy is database proxy that de-identifies PII for PostgresDB and MySQL

Surf Surf is a database proxy that is capable of de-identifying PII and anonymizing sentive data fields. Supported databases include Postgres, MySQL,

null 1 Dec 14, 2021
[mirror] the database client and tools for the Go vulnerability database

The Go Vulnerability Database golang.org/x/vulndb This repository is a prototype of the Go Vulnerability Database. Read the Draft Design. Neither the

Go 201 Nov 25, 2022
Database - Example project of database realization using drivers and models

database Golang based database realization Description Example project of databa

Denis 1 Feb 10, 2022
a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

siddontang 3.9k Dec 8, 2022
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.9k Dec 7, 2022
A high-performance MySQL proxy

kingshard 中文主页 Overview kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read

Fei Chen 6.1k Dec 6, 2022
Golang MySql binary log replication listener

Go MySql binary log replication listener Pure Go Implementation of MySQL replication protocol. This allow you to receive event like insert, update, de

Pavel <Ven> Gulbin 189 Oct 25, 2022
MySQL replication topology management and HA

orchestrator [Documentation] orchestrator is a MySQL high availability and replication management tool, runs as a service and provides command line ac

null 4.7k Dec 1, 2022
db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

null 25 Nov 17, 2022
一个使 mysql,pgsql 数据库表自动生成 go struct 的工具

db2go 一个使 mysql、pgsql 数据库表自动生成 go struct 的工具 快速使用 将项目放入到GOPATH/src目录下

易水韩 19 Nov 25, 2022