Vitess is a database clustering system for horizontal scaling of MySQL.

Overview

Maven Central Build Status codebeat badge Go Report Card FOSSA Status CII Best Practices Coverage

Vitess

Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding.

By encapsulating shard-routing logic, Vitess allows application code and database queries to remain agnostic to the distribution of data onto multiple shards. With Vitess, you can even split and merge shards as your needs grow, with an atomic cutover step that takes only a few seconds.

Vitess has been a core component of YouTube's database infrastructure since 2011, and has grown to encompass tens of thousands of MySQL nodes.

For more about Vitess, please visit vitess.io.

Vitess has a growing community. You can view the list of adopters here.

Reporting a Problem, Issue, or Bug

To report a problem, the best way to get attention is to create a GitHub issue using proper severity level based on this guide.

For topics that are better discussed live, please join the Vitess Slack workspace. You may post any questions on the #general channel or join some of the special-interest channels.

Follow Vitess Blog for low-frequency updates like new features and releases.

Security

Reporting Security Vulnerabilities

To report a security vulnerability, please email vitess-maintainers.

See Security for a full outline of the security process.

Security Audit

A third party security audit was performed by Cure53. You can see the full report here.

License

Unless otherwise noted, the Vitess source files are distributed under the Apache Version 2.0 license found in the LICENSE file.

FOSSA Status

Issues
  • workflow: Add horizontal resharding workflow.

    workflow: Add horizontal resharding workflow.

    This is a beta version implementation, which simply supports creating the horizontal workflow through UI and updating progress of each step.

    In the end-to-end test (test/workflow_horizontal_resharding.py), only happy path is tested. To simplify the test, rather than fetching information from front-end and checking explicitly, I manually interacting with the UI after setting up the environments.

    Unit test will be added later.

    Possible Optimizations (wait for feedback from reviewers, the listed points might be useless):

    • make UI more friendly, s.t. the user input doesn't have to input the shard list
    • supports action for the workflow, which allows the user to stop and restart the workflow
    • adding possible corner case end-to-end test

    This change is Reviewable

    opened by wangyipei01 87
  • Experimental: automated, scheduled, dependency free online DDL via gh-ost/pt-online-schema-change

    Experimental: automated, scheduled, dependency free online DDL via gh-ost/pt-online-schema-change

    This PR (work in progress) introduces zero dependency online schema changes with gh-ost/pt-online-schema-change.

    UPDATE: this comment edited to reflect support for pt-online-schema-change. Originally this PR only supported gh-ost. Mostly whenever you see gh-ost, consider pt-online-schema-change to apply, as well.

    TL;DR

    User will issue:

    alter with 'gh-ost' table example modify id bigint not null;
    
    alter with 'pt-osc' table example modify id bigint not null
    

    or

    $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global \
        ApplySchema -sql "alter with 'gh-ost' table example modify id bigint unsigned not null" commerce
    
    $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global \
        ApplySchema -sql "alter with 'pt-osc' table example modify id bigint unsigned not null" commerce
    

    and vitess will schedule an online schema change operation to run on all relevant shards, then proceed to apply the change via gh-ost on all shards.

    While this PR is WIP, this flow works. More breakdown to follow, indicating what's been done and what's still missing.

    The ALTER TABLE problem

    First, to iterate the problem: schema changes have always been a problem with MySQL; a straight ALTER is a blocking operation; a ONLINE ALTER is only "online" on the master/primary, but is effectively blocking on replicas. Online schema change tools like pt-online-schema-change and gh-ost overcome these limitations by emulating an ALTER on a "ghost" table, which is populated from the original table, then swapped in its space.

    For disclosure, I authored gh-ost's code as part of the database infrastructure team at GitHub.

    Traditionally, online schema changes are considered to be "risky". Trigger based migrations add significant load onto the master server, and their cut-over phase is known to be a dangerous point. gh-ost was created at GitHub to address these concerns, and successfully eliminated concerns for operational risks: with gh-ost the load on the master is low, and well controlled, and the cut-over phase is known to cause no locking issues. gh-ost comes with different risks: it applies data changes programmatically, thus the issue of data integrity is of utmost importance. Another note of concern is data traffic: going out from MySQL into gh-ost and back into MySQL (as opposed to all-in MySQL in pt-online-schema-change).

    This way or the other, running an online schema change is typically a manual operation. A human being will schedule the migration, kick it running, monitor it, possibly cut-over. In a sharded environment, a developer's request to ALTER TABLE explodes to n different migrations, each needs to be scheduled, kicked, monitored & tracked.

    Sharded environments are obviously common for vitess users and so these users feel the pain more than others.

    Schema migration cycle & steps

    Schema management is a process that begins with the user designing a schema change, and ends with the schema being applied in production. This is a breakdown of schema management steps as I know them:

    1. Design code
    2. Publish changes (pull request)
    3. Review
    4. Formalize migration command (the specific ALTER TABLE or pt-online-schema-change or gh-ost command)
    5. Locate: where in production should this migration run?
    6. Schedule
    7. Execute
    8. Audit/monitor
    9. Cut-over/complete
    10. Cleanup
    11. Notify user
    12. Deploy & merge

    What we propose to address

    Vitess's architecture uniquely positions it to be able to automate away much of the process. Specifically:

    • Formalize migration command: turning an ALTER TABLE statement into a gh-ost invocation is super useful if done by vitess, since vitess can not only validate schema/params, but also can provide credentials, identify a throttle-control replica, can instruct gh-ost on how to communicate progress via hooks, etc.
    • Locate: given schema/table, vitess just knows where the table is located. It knows if the schema is sharded. It knows who the shards are, who the shards masters are. It knows where to run gh-ost. Last, vitess can tell us which replicas we can use for throttling.
    • Schedule: vitess is again in a unique position to schedule migrations. The fact someone asks for a migration to run does not mean the migration should start right away. For example, a shard may already be running an earlier migration. Running two migrations at a time is less than ideal, and it's best to wait out the first migration before beginning the second. A scheduling mechanism is both useful to running the migrations in optimal order/sequence, as well as providing feedback to the user ("your migration is on hold because this and that", or "your migration is 2nd in queue to run")
    • Execute: vttablet is the ideal entity to run a migration; can read instructions from topo server and can write progress to topo server. vitess is aware of possible master failovers and can request a re-execute is a migration is so interrupted mid process.
    • Audit/monitor: vtctld API can offer endpoints to track status of a migration (e.g. "in progress on -80, in queue on 80-"). It may offer progress pct and ETA.
    • cut-over/complete: in my experience with gh-ost, the cut-over phase is safe to automate away.
    • cleanup: the old table needs to be dropped; vttablet is in an excellent position to automate that away.

    What this PR does, and what we expect to achieve

    The guideline for this PR is: zero added dependencies; everything must be automatically and implicitly available via a normal vitess installation.

    A breakdown:

    User facing

    This PR enables the user to run an online schema migration (aka online DDL) via:

    • vtgate: the user connects to vitess with their standard MySQL client, and issues a ALTER WITH 'gh-ost' TABLE ... statement. Notice this isn't a valid MySQL syntax -- it's a hint for vitess that we want to run this migration online. vitess still supports synchronous, "normal" ALTER TABLE statements, which IMO should be discouraged.
    • vtctl: the user runs vtctl ApplySchema -sql "alter with _gh-ost' table ...".

    The response, in both cases, is a migration ID, or a job ID, if you will. Consider the following examples.

    via vtgate:

    
    mysql> create table example(id int auto_increment primary key, name tinytext);
    
    mysql> show create table example \G
    
    CREATE TABLE `example` (
      `id` int NOT NULL AUTO_INCREMENT,
      `name` tinytext,
      PRIMARY KEY (`id`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    
    mysql> alter with 'gh-ost' table example modify id bigint not null, add column status int, add key status_dx(status);
    +--------------------------------------+
    | uuid                                 |
    +--------------------------------------+
    | 211febfa-da2d-11ea-b490-f875a4d24e90 |
    +--------------------------------------+
    
    -- <wait...>
    
    mysql> show create table example \G
    
    CREATE TABLE `example` (
      `id` bigint NOT NULL,
      `name` tinytext,
      `status` int DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `status_dx` (`status`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    

    via vtctl:

    $ mysql -e "show create table example\G"
    
    CREATE TABLE `example` (
      `id` bigint NOT NULL,
      `name` tinytext,
      `status` int DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `status_dx` (`status`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    
    
    $ vtctl -topo_implementation etcd2 -topo_global_server_address localhost:2379 -topo_global_root /vitess/global \
        ApplySchema -sql "alter with 'gh-ost'  table example modify id bigint unsigned not null" commerce
    8ec347e1-da2e-11ea-892d-f875a4d24e90
    
    
    $ mysql -e "show create table example\G"
    
    CREATE TABLE `example` (
      `id` bigint unsigned NOT NULL,
      `name` tinytext,
      `status` int DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `status_dx` (`status`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
    

    In both cases, a UUID is returned, which can be used for tracking (WIP) the progress of the migration across shards.

    Parser

    Vitess' parser now accepts ALTER WITH 'gh-ost' TABLE and ALTER WITH 'pt-osc' TABLE syntax. We're still to determine if this is the exact syntax we want to go with.

    Topo

    Whether submitted by vtgate or vtctl, we don't immediately run the migration. As mentioned before, we may wish to postpone the migration. Perhaps the relevant servers are already running a migration.

    Instead, we write the migration request into global topo, e.g.:

    • key: /vitess/global/schema-migration/requests/90c5afd4-da38-11ea-a3ff-f875a4d24e90
    • content:
    {"keyspace":"commerce","table":"example","sql":"alter table example modify id bigint not null","uuid":"90c5afd4-da38-11ea-a3ff-f875a4d24e90","online":true,"time_created":1596701930662801294,"status":"requested"}
    

    Once we create the request in topo, we immediately return the generated UUID/migration ID (90c5afd4-da38-11ea-a3ff-f875a4d24e90 in the above example) to the user.

    vtctld

    vtctld gets a conceptual "upgrade" with this PR. It is no longer a reactive service. vtctld now actively monitors new schema-migration/requests in topo.

    ~~When it sees such a request, it evaluates what are the relevant n shards.~~

    ~~With current implementaiton, it writes n "job" entries, one per shard. e.g.~~

    • /vitess/global/schema-migration/jobs/commerce/-80/ce45b84a-da2d-11ea-b490-f875a4d24e90 and /vitess/global/schema-migration/jobs/commerce/80-/ce45b84a-da2d-11ea-b490-f875a4d24e90 for a keyspace with two shards; or just
    • /vitess/global/schema-migration/jobs/commerce/0/1dd17132-da23-11ea-a3d2-f875a4d24e90 for a keyspace with one shard.

    DONE: WIP: we will investigate use of new VExec to actually distribute the jobs to vttablet.

    what vtctld does now, is, once it sees a migration request, it pushes a VExec request for that migration. If the VExec request succeeds, that means all shards have been notified, and vtctld can stow away the migration request (work is complete as far as vtctld is concerned). If VExec returns with an error, that means at least one shard did not get the request, and vtctld will keep retrying pushing this request.

    vttablet

    This is where most of the action takes place.

    vttablet runs a migration service which continuously probes for, schedules, and executes migrations.

    DONE: ~~With current implementation, tablets which have tablet_type=MASTER continuously probe for new entries. We look to replace this with VExec.~~

    migration requests are pushed via VExec; the request includes the INSERT IGNORE query that persists the migration in _vt.schema_migrations. The tablet no longer reads from, nor writes to, Global Topo.

    A new table is introduced: _vt.schema_migrations, which is how vttablet manages and tracks its own migrations.

    vttablet will only run a single migration at a time.

    vttablet will see if there's an unhandled migration requests. It will queue it.

    vttablet will make a migration ready if there's no running migration and no other migration is marked as ready.

    vttablet will run a ready migration. This is really the interesting part, with lots of goodies:

    • vttablet will evaluate the gh-ost ... command to run. It will obviously populate --alter=... --database=....
    • vttablet creates a temp directory where it generates a script to run gh-ost.
    • vttablet creates a hooks path and auto-generates hook files. The hooks will interact with vttablet
    • vttablet has an API endpoint by which the hooks can communicate gh-ost's status (started/running/success/failure) with vttablet.
    • vttablet provides gh-ost with --hooks-hint which is the migration's UUID.
    • vttablet automatically generates a gh-ost user on the MySQL server, with a random password. The password is never persisted and does not appear on ps. It is written to, and loaded from, an environment variable.
    • vttablet grants the proper privileges on the newly created account
    • vttablet will destroy the account once migration completes.
    • vitess repo includes a gh-ost binary. We require gh-ost from openark/gh-ost as opposed to github/gh-ost because we've had to make some special adjustments to gh-ost s oas to support this flow. I do not have direct ownership to github/gh-ost and cannot enforce those changes upstream, though I have made the contribution requestss upstream.
    • make build automatically appends gh-ost binary, compressed, to vttablet binary, via Ricebox.
    • vttablet, upon startup, auto extracts gh-ost binary into /tmp/vt-gh-ost. Please note that the user does not need to install gh-ost.
    • WIP: vttablet to report back the job as complete/failed. We look to use VExec. TBD.

    Tracking breakdown

    • [x] New OnlineDDL struct, defines a migration request and its status
    • [x] Parser supports ALTER WITH 'gh-ost' TABLE and ALTER WITH 'pt-osc' TABLE syntax
    • [x] builder and analyzer to create an Online DDL plan (write to topo)
    • [x] vtctl to skip "big changes" check when -online_schema_change is given
    • [x] tablet_executor to submit an online DDL request to topo as opposed to running it on tablets
    • [x] vtctld runs a daemon to monitor for, and review migration requests
    • [x] vtctld evaluates which shards are affected
    • [x] _vt.schema_migrations backend table to support migration automation (on each shard))
    • [x] vttablet validates MySQL connection and variables
    • [x] vttablet creates migration command
    • [x] vttablet creates hooks
    • [x] vttablet provides HTTP API for hooks to report their status back
    • [x] vttablet creates gh-ost user with random password
    • [x] vttablet destroys gh-ost user upon completion
    • [x] gh-ost embedded in vttablet binary and auto-extracted by vttablet
    • [x] vttablet runs a dry-run execution
    • [x] vttablet runs a --execute (actual) execution
    • [x] vttablet supports a Cancel request (not used yet) to abort migration
    • [x] vttablet as a state machine to work throught the migration steps
    • [x] counters for gh-ost migration requests, suceessful and failed migrations
    • [x] use of VExec to apply migrations onto tablets
    • [x] use of VExec to control migrations (abort, retry)
    • [ ] consider flow for retries
    • [ ] identify a reparent operation that runs during a migration, probabaly auto-restart the migration
    • [ ] vttablet to heuristically check for available disk space
    • [x] tracking, auditing of all migrations
    • [x] getting gh-ost logs if necessary
    • [x] what's the best way to suggest we want an online migration? Does current ALTER WITH 'gh-ost' TABLE... and ALTER WITH 'pt-osc' TABLE syntax make sense? Other?
    • [ ] For first iteration, migrations and Reshard operations should be mutually exclusive. Can't run both at the same time. Next iterations will remove this constraint.
    • [x] ~~throttle by replica~~
    • [ ] ~~wait for replica to catch up with new credentials before starting the migration~~
    • [x] Use vttablet throttler
    • [x] pt-online-schema-change bundled inside vttablet binary
    • [x] support pt-online-schema-change
    • [ ] ~~define foreign key flags for pt-online-schema-change execution~~ - user can define as runtime flags
    • [x] clenaup online-ddl directory after success
    • [ ] control throttling
    • [x] control termination (panic abort)
    • [x] control termination (panic abort) even after vttablet itself crashes
    • [x] pt-online-schema-change passwords are in cleartext. Can we avoid that?
    • [x] vtctl ApplySchema use same WITH 'gh-ost' and WITH 'pt-osc' query hints as in vtgate.
    • [x] support override of gh-ost and pt-online-schema-change paths
    • [x] cleanup pt-osc triggers after migration failure
    • [x] forcibly remove pt-osc triggers on migration cancellation (overlaps with previous bullet, but has stronger guarantee)
    • [x] cleanup pt-osc triggers from stale/zombie pt-osc migration
    • [x] vtctl OnlineDDL command for simple visibility and manipulation. See https://github.com/vitessio/vitess/pull/6547#issuecomment-681879259
    • [x] end to end tests
    • [x] populate artifacts column, suggesting which tables need to be cleaned up after migration

    Quite likely more entries to be added.

    Further reading, resources, acknowledgements

    We're obviously using gh-ost. I use my own openark/gh-ost since I have no ownership of the original https://github.com/github/gh-ost. gh-ost was/is developed by GitHub 2016-2020.

    pt-online-schema-change is part of the popular Percona Toolkit

    The schema migratoin scheduling and tracking work is based on my previous work at GitHub. The implementation in this PR is new and rewritten, but based on concepts that have matured on my work on skeefree. Consider these resources:

    Also:

    • An early presentation on gh-ost

    Initial incarnation of this PR: https://github.com/planetscale/vitess/pull/67; some useful comments on that PR.

    Call for feedback

    We're looking for community's feedback on the above suggestions/flow. Thank you for taking the time to read and respond!

    Type: Enhancement 
    opened by shlomi-noach 65
  • Adding TLS support to the Java client and JDBC driver

    Adding TLS support to the Java client and JDBC driver

    re: Issue #2209

    TLS connections are already supported by VTGate, gRPC, and all of the non-Java clients. For Java to reach parity, support is needed in the grpc-client, client, and jdbc modules under src/github.com/youtube/vitess/java.

    This PR is a work in progress... visible for early review and vetting, but not yet ready to merge.

    At this point, the VitessJDBCUrl class parses an optional parameter, which indicates that a connection should use SSL. The parameter follows standard MySQL JDBC conventions.

    Pending commits will:

    1. Update GrpcClientFactory, to expose a method for creating a TLS-enabled RpcClient in addition to the current plaintext one.
    2. Update VitessVTGateManager, to use this new method when the JDBC URL calls for SSL.
    3. Write unit test coverage for the above, using a vtgateclienttest mock in a style similar to the src/github.com/youtube/vitess/test/encrypted_transport.py script.
    4. End-to-end testing. Update examples/local/vtgate-up.sh, to use TLS with a self-signed cert when an optional parameter is passed. Also update the examples/client_jdbc.sh test script, to likewise use a TLS connection when an optional parameter is passed.

    Any feedback about this approach is definitely welcomed!


    This change is Reviewable

    opened by steve-perkins 63
  • WIP - Heartbeat support in vttablet

    WIP - Heartbeat support in vttablet

    I'm creating tests and testing in my environment. Submitting PR to see what travis has to say.

    Any comments welcome.


    This change is Reviewable

    opened by bbeaudreault 37
  • Single shard targeting in V3

    Single shard targeting in V3

    There is no way to target a single shard in the V3 APIs.

    We need to target single shards for two reasons:

    1. We want to shard some of our processing in that we have jobs that process an entire single shard at the time.
    2. There's a very large number of scatter/gather functionality that is currently not supported (e.g. limit on joins, order by and so on). Some we can work around on the query level but some we have to build our own scatter/gather logic.

    Item 2 could eventually be addressed by extending scatter/gather in V3 but item 1 can never be addressed by V3.

    opened by tirsen 31
  • Remove unnecessary blocking in stats package.

    Remove unnecessary blocking in stats package.

    @alainjobart @sougou

    Otherwise it imposes global locking in VTGate.


    This change is Reviewable

    opened by guoliang100 31
  • helm: Allow non-leader Orchestrator instances to accept requests

    helm: Allow non-leader Orchestrator instances to accept requests

    Orchestrator 3.0.7 added a proxy that forwards master only requests, so we don’t have to workaround that by having perpetually unready pods via the /api/leader-check endpoint

    cc @shlomi-noach @enisoc

    opened by derekperkins 30
  • easier build

    easier build

    I worked on this because I finished a brew formula to install Vitess on macOS with one command: brew install vitess. Also, there was some conversation on the Slack channel to improve the build process and make it easier.

    The main change in here is: we will push the vendor libs, instead of getting them during the bootstrap.sh script run. Advantages of this approach:

    • We ensure the version of our libs, not just by the vendor.json file, but they actually will follow the history of this repo.
    • Makes the build process a lot easier. With this, I can build Vitess without needing to run boostrap.sh. This is useful for folks that don't need to change the code but just want to build it.
    • Homebrew build process doesn't allow us to do go get during it, as that is insecure. But, as we vendor our dependencies we don't need to go get during build time anymore.

    Drawbacks:

    • when changing the vendor.json file, we need to run go vendor sync or the bootstrap script again, to upload the new version of the lib.
    • This repository will get bigger in git size. (the local size will still be the same, though)

    I think the advantages outweighs the disadvantages, as the ratio of local builds vs changing vendor.json file is bigger.

    At, least at GitHub, thats what we do, we vendored and upload the lib sources to git. Also, open source projects such as Consul do the same.

    Thoughts? cc @sougou @vmg

    opened by arthurnn 30
  • Adds table name to the field proto

    Adds table name to the field proto

    https://dev.mysql.com/doc/refman/5.7/en/c-api-data-structures.html

    The table and table_length fields are available in the C api but are not passed through to clients for use. One of our core java client libraries depends on http://docs.oracle.com/javase/7/docs/api/java/sql/ResultSetMetaData.html#getTableName(int), which in the existing Vitess Driver was hardcoded to return null.

    This PR adds table to the Field proto, populates it in the mysql.Fields() function, and allows access in the Java client. All tests updated to account for the new field, and succeeding.

    I've also tested this in our deployment. However, I've not been able to get the full make test to work in my local dev yet, this was tested with make site_test.

    This is my first foray into both go and vitess, so feedback welcome.

    opened by bbeaudreault 28
  • Preserve case of column names

    Preserve case of column names

    When Vitess receives a query, the tokenizer lowercases all column names so they can be string-compared safely throughout the system. This means all column names returned in the eventual query result are lowercase as well. This causes a problem in PHP when users expect to look up columns by name with the same case they requested in the original query:

    $result = query('select Column from table');
    $row = $result->fetch();
    echo $row['Column']; // array key doesn't exist
    echo $row['column']; // works
    

    In Java (JDBC) they address this by specifying that column name lookups are case-insensitive. We recently fixed our Java client to conform to this (https://github.com/youtube/vitess/pull/1572).

    We could do something similar in PHP by implementing ArrayAccess or extending ArrayObject. Each of those has its pros and cons, but either way, the object we return would not be a true array, so things like array_keys() wouldn't work.

    @tslater @Rastusik Can you weigh in on whether this makes sense from a PHP app writer's perspective?

    @sougou

    Type: Feature Request 
    opened by enisoc 27
  • Fix schema engine reload so it accounts for swapped tables during online DDL

    Fix schema engine reload so it accounts for swapped tables during online DDL

    Description

    This PR addresses the following bug in schema engine reload process:

    • The schema engine depends on the information_schema.tables create_time field to tell when a table is updated. Unlike the name might indicate, this field is updated when a structural DDL is performed on a table (e.g. a column add/delete).
    • As an optimization, the schema engine keeps track of the last time it loaded (reloaded) the schema, and saves this timestamp (called lastChange).
    • When reloading schema, for each table we then check the create_time for that table against this saved timestamp, and if it's older, we skip examining the table further for changes.
    • Vitess online DDL swaps the tables, and a table swap does not affect the create_time timestamp for the table (it remains the original timestamp of when online DDL created the temporary table). If after this, but before the online DDL is done and the tables are swapped, a schema reload is run (in any of the various ways), we will record all the changes in schema at that point, and save the current time in the schema engine's lastChange field.
    • Now, after the online DDL completes, a schema reload is internally signaled:
      • For the table we are interested in (the one that was just DDL-ed), the create_time field is now older than the schema engine lastChange field. We therefore do not update the schema engine's view of this table's schema.

    Changes:

    • Instead of keeping track of the lastChange time in schema engine, we track the create_time of every table explicitly. On a subsequent refresh, if we find the create_time of the table reported by MySQL differs from what we have cached in-memory, then we will assume the table has changed and reload its schema.
    • This will address the issue with tables being swapped during OnlineDDL without the create_time being modified.

    Related Issue(s)

    Fixes #9287

    Checklist

    • [ ] Should this PR be backported?
    • [x] Tests were added or are not required
    • [x] Documentation was added or is not required

    Deployment Notes

    opened by mvh-stripe 0
  • [vtadmin-api] /api/tablet(s) missing data for RESTORE tablets

    [vtadmin-api] /api/tablet(s) missing data for RESTORE tablets

    The /api/tablet(s) endpoints do not include the tablet type for RESTORE tablets. (Internal data redacted.)

    {
      "result": {
        "cluster": {
          "id": "...",
          "name": "..."
        },
        "tablet": {
          "alias": {
            "cell": "...",
            "uid": 0, 
          },
          "hostname": "..."
        },
        "state": 2,
        "FQDN": "..."
      },
      "ok": true
    }
    

    At first blush, it seems like this might be because RESTORE corresponds to 0 in the TabletType enum but it's actually... 7? 🧐

    Note that keyspace and shard are missing, too.

    Type: Bug Component: VTAdmin 
    opened by doeg 0
  • [vtadmin-web] Consolidate font sizing into tailwind.config.js

    [vtadmin-web] Consolidate font sizing into tailwind.config.js

    Description

    This consolidates all of our font sizing variables into the tailwind.config.js file. Since we use a 10px basis, as opposed to the tailwind/browser default of 16px, our font scale differs from the tailwind docs, hence the overrides. (This will also make it 1000x easier to adjust font sizes if we do ever decide to change the rem basis.)

    Also added some lorem ipsum to the /debug page until we get Storybook set up.

    Related Issue(s)

    • #9215

    Checklist

    • [ ] Should this PR be backported? No
    • [x] Tests were added or are not required
    • [x] Documentation was added or is not required

    Deployment Notes

    N/A

    Type: Internal Cleanup Component: VTAdmin release notes none 
    opened by doeg 0
  • vtctl ApplySchema accepts 'ALTER VITESS_MIGRATION...' statements

    vtctl ApplySchema accepts 'ALTER VITESS_MIGRATION...' statements

    Description

    The following is now supported:

    $ vtctlclient ApplySchema -sql "alter vitess_migration 'dce5f77e_51de_11ec_a049_0a43f95f28a3' cancel" commerce
    

    ApplySchema now accepts all variants of ALTER VITESS_MIGRATION....

    Why are we doing this?

    We long since wanted to deprecate VExec for OnlineDDL. Right now vtctl OnlineDDL <command> heavily relies on wrangler/VExec to get the command to the underlying tablets. However, ApplySchema already has the required logic to get in to the QueryService. We will reimplement vtctl OnlineDDL on top of ApplySchema instead of VExec.

    This PR is a first step towards that re-implementation.

    Related Issue(s)

    • Tracking: https://github.com/vitessio/vitess/issues/6926#issuecomment-817619302
    • https://github.com/vitessio/vitess/pull/8971 as a previous attempt, reverted since.

    Checklist

    • [ ] Should this PR be backported?
    • [ ] Tests were added or are not required
    • [ ] Documentation was added or is not required
    Type: Enhancement Component: Cluster management release notes 
    opened by shlomi-noach 0
  • Allow unowned lookup vindex columns to be `null`

    Allow unowned lookup vindex columns to be `null`

    Description

    This change allows columns for unowned lookup vindexes to be null.

    Related Issue(s)

    N/A

    Checklist

    • [ ] Should this PR be backported?
    • [ ] Tests were added or are not required
    • [ ] Documentation was added or is not required
    opened by arthurschreiber 0
  • fix: avoid parenthesis around default null in BLOB, TEXT, GEOMETRY and JSON column

    fix: avoid parenthesis around default null in BLOB, TEXT, GEOMETRY and JSON column

    Description

    This PR avoids adding parenthesis around null value in default for BLOB, TEXT, GEOMETRY and JSON column.

    Related Issue(s)

    • Fixes #9297

    Checklist

    • [X] Tests were added or are not required
    • [X] Documentation was added or is not required

    Deployment Notes

    Type: Bug Component: Query Serving release notes 
    opened by harshit-gangal 0
  • Upgrade Downgrade Testing

    Upgrade Downgrade Testing

    Description

    This pull request introduces a more comprehensive upgrade/downgrade test strategy in order to comply with VEP-3. The new strategy is thoroughly discussed in https://github.com/vitessio/vitess/issues/7344 (specifically in this comment: https://github.com/vitessio/vitess/issues/7344#issuecomment-949297139).

    Checklist

    • [ ] Should this PR be backported?
    • [ ] Tests were added or are not required
    • [ ] Documentation was added or is not required

    Deployment Notes

    Component: Build/CI Type: CI/Build release notes 
    opened by frouioui 0
  • Change heartbeat writer to use vt_app user

    Change heartbeat writer to use vt_app user

    Description

    As described in the issue, we do not want the heartbeat writer to be using the vt_dba user which has super privileges. This creates issues with PRS wherein, the insert commands succeed even after the mysql instance has been set to read only causing the old primary to diverge from the new primary.

    This PR changes the user back to vt_app.

    Related Issue(s)

    • Fixes #9290

    Checklist

    • [ ] Should this PR be backported?
    • [x] Tests were added or are not required
    • [x] Documentation was added or is not required

    Deployment Notes

    Type: Bug Component: Cluster management release notes none 
    opened by GuptaManan100 2
  • Online DDL: vtctl OnlineDDL supports 'complete' command

    Online DDL: vtctl OnlineDDL supports 'complete' command

    Description

    Followup to https://github.com/vitessio/vitess/pull/9171 In https://github.com/vitessio/vitess/pull/9171 we introduced ALTER VITESS_MIGRATION '...' COMPLETE command. This PR complements the SQL interface by adding a vtctl OnlineDDL <keyspace> complete <uuid> command.

    To be documented.

    Related Issue(s)

    Tracking: #6926

    Checklist

    • [ ] Should this PR be backported?
    • [ ] Tests were added or are not required
    • [ ] Documentation was added or is not required
    Type: Enhancement Component: Cluster management release notes 
    opened by shlomi-noach 0
  • `CREATE TABLE` statements use format unsupported on MySQL < 8.x

    `CREATE TABLE` statements use format unsupported on MySQL < 8.x

    Overview of the Issue

    In #9011, the SQL generated for CREATE TABLE statements was changed to generate expression statements for default values:

    create table t2 (b blob default 'abc');
    

    was changed to:

    create table t2 (b blob default ('abc'));
    

    Unfortunately, this syntax is only supported on MySQL 8.0 and later, and is unsupported on earlier MySQL versions like 5.7.

    mysql> create table t2 (b blob default ('abc'));
    ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '('abc'))' at line 1
    

    Vitess should not translate between one or the other, and just pass through whatever syntax the user is passing in.

    Reproduction Steps

    Steps to reproduce this issue:

    1. Run a version of Vitess that includes the changes from #9011 on MySQL 5.7:

    2. Try creating the following table:

      create table t2 (b blob default 'abc');
      
    opened by arthurschreiber 5
Releases(v12.0.0)
  • v12.0.0(Oct 26, 2021)

    Release of Vitess v12.0.0

    Major Changes

    This release includes the following major changes or new features.

    Inclusive Naming

    A number of CLI commands and vttablet RPCs have been deprecated as we move from master to primary. All functionality is backwards compatible except as noted under Incompatible Changes. Deprecated commands and flags will be removed in the next release (13.0). Deprecated vttablet RPCs will be removed in the subsequent release (14.0).

    Gen4 Planner

    The newest version of the query planner, Gen4, becomes an experimental feature as part of this release. While Gen4 has been under development for a few release cycles, we have now reached parity with its predecessor, v3.

    To use Gen4, VTGate's -planner_version flag needs to be set to Gen4Fallback.

    Query Buffering during failovers

    In order to support buffering during resharding cutovers in addition to primary failovers, a new implementation of query buffering has been added. This is considered experimental. To enable it the flag buffer_implementation should be set to keyspace_events. The existing implementation (flag value healthcheck) will be deprecated in a future release.

    Incompatible Changes

    CLI command output

    Wherever CLI commands produced master or MASTER for tablet type, they now produce primary or PRIMARY. Scripts and tools that depend on parsing command output will need to change.

    Example:

    $ vtctlclient -server localhost:15999 ListAllTablets 
    zone1-0000000100 sourcekeyspace 0 primary 192.168.0.134:15100 192.168.0.134:17100 [] 2021-09-24T01:12:00Z
    zone1-0000000101 sourcekeyspace 0 rdonly 192.168.0.134:15101 192.168.0.134:17101 [] <null>
    zone1-0000000102 sourcekeyspace 0 replica 192.168.0.134:15102 192.168.0.134:17102 [] <null>
    zone1-0000000103 sourcekeyspace 0 rdonly 192.168.0.134:15103 192.168.0.134:17103 [] <null>
    

    Default Behaviour Changes

    Enabling reserved connections

    Use of reserved connections is controlled by the vtgate flag -enable_system_settings. This flag has in the past defaulted to false (or OFF) in release versions (i.e. x.0 and x.0.y) of Vitess, and to true (or ON) in development versions.

    From Vitess 12.0.0 onwards, it defaults to true in all release and development versions. You can read more about this here. Hence you should specify this flag explicitly, so that you are sure whether it is enabled or not, regardless of which Vitess release/build/version you are running.

    If you have reserved connections disabled, you will get the old Vitess behavior: where most system settings (e.g. sql_mode) are just silently ignored by Vitess. In situations where you know your backend MySQL defaults are acceptable, this may be the correct setting to ensure the best possible performance of the vttablet <-> MySQL connection pools. This comes down to a trade-off between compatibility and performance/scalability. You should also review this section of the documentation when deciding whether or not to enable reserved connections.

    Deprecations

    CLI commands

    VExec is deprecated and removed. All Online DDL commands should be run through OnlineDDL.

    OnlineDDL revert is deprecated. Use REVERT VITESS_MIGRATION '...' SQL command either via ApplySchema or via vtgate.

    InitShardMaster is deprecated, use InitShardPrimary instead.

    SetShardIsMasterServing is deprecated, use SetShardIsPrimaryServing instead.

    Various command flags have been deprecated and new variants provided.

    • DeleteTablet flag allow_master replaced with allow_primary
    • PlannedReparentShard flag avoid_master replaced with avoid_tablet
    • PlannedReparentShard flag new_master replaced with new_primary
    • BackupShard flag allow_master replaced with allow_primary
    • Backup flag allow_master replaced with allow_primary
    • EmergencyReparentShard flag new_master replaced with new_primary
    • ReloadSchemeShard flag include_master replaced with include_primary
    • ReloadSchemaKeyspace flag include_master replaced with include_primary
    • ValidateSchemaKeyspace flag skip-no-master replaced with skip-no-primary

    Minor Changes

    Query Serving

    • Add SHOW VITESS_REPLICATION_STATUS Query Support #8900
      • The new SHOW VITESS_REPLICATION_STATUS command/query shows the MySQL replica/replication (not vreplication) health for the vitess deployment.

    Governance

    • Added @GuptaManan100 to maintainers #8833

    Changelog

    Bug fixes

    Build/CI

    • Update golang/x dependencies #8688
    • tests: use AtomicInt32 instead of int to fix races #8696

    Cluster management

    • Use consul lock properly #8310
    • Trivial: mysqlctl reinit_config -h was not showing help #8369
    • Port #8422 to main branch #8745
    • [vtctl] Add missing call to flag.Parse in commandGetRoutingRules #8795
    • Fix for padding in OrderAndCheckPartitions #8873
    • tabletmanager: correct implementations of deprecated RPCs for backwards compatibility #9059

    General

    • Avoid some pollution of CLI argument namespace across vtgate and vttablet #9026

    Observability

    • schema engine: reset table size stats when a table is dropped #8634

    Query Serving

    • Fix for function calls in DEFAULT value of CREATE TABLE statement in main #8477
    • Fixing multiple issues related to onlineddl/lifecycle #8500
    • boolean values should not be parenthesised in default clause #8502
    • mysql/server: abort connection if secure transport is required #8504
    • Clear only the current shard's schemacopy rows during schema reload #8519
    • go/mysql/auth: fix error message for wrong auth method. #8525
    • Fix handshake protocol with MySQL client #8537
    • Copy parameter data from byte buffer in COM_STMT_SEND_LONG_DATA #8562
    • Negative float default #8587
    • [vtexplain] prevent panic on show tables/show full tables #8601
    • OnlineDDL: better scheduling/cancellation logic #8603
    • Fix parsing of FIRST clause in ALTER TABLE #8614
    • onlineddl Executor: build schema with DBA user #8624
    • Handle subquery merging with references correctly #8662
    • Fixing error on deletes from owning table for a lookup being populated by CreateLookupVindex after a SwitchWrites #8701
    • Fixing a panic in vtgate with OLAP mode #8722
    • default to primary tablet if not set in VStream api #8755
    • Fix bug in parsing of version comments #8770
    • Proper merge of the SysTableTableName fields when joining two routes #8771
    • fix parsing of MySQL Server Version flag #8791
    • Fix typo in pool config syntax. Fixes #8819 #8820
    • VtGate schema tracker to use provided user #8857
    • fix merging dba query as a subquery both in v3 and gen4 #8871
    • sql_select_limit support #9027
    • [12.0] Handle sql_mode setting to compare modes in any order #9041
    • [12.0] Fix for schema tracker not recognizing RENAME for schema reloading #9043
    • [12.0] Fix savepoint support with reserved connections #9047
    • [12.0] Gen4: Handle single column vindex correctly with multi column #9061

    VReplication

    • MoveTables: don't create unnecessary streams on the target for non-intersecting sources and targets #8090
    • VDiff: Add BIT datatype to list of byte comparable types #8401
    • Fix vreplication error metric #8483
    • Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
    • Fix VReplication logging to file and db #8521
    • Fix passing the wrong cell/cells variable to CreateLookupVindex #8590
    • Refresh SrvVSchema after an ExternalizeVindex: was missing #8670
    • Fixing missing argument in function call #8692
    • Fix how we identify MySQL generated columns #8763

    VTAdmin

    • [vtadmin-web] Set Backup status indicators to the correct colours #8410
    • [vtadmin] Fix bad copy-paste in pool config flag parsing #8518

    vtctl

    • Print actual current datetime for vtctl dry run commands #8778

    vttestserver

    • Remove lingering mysqld sock files on vttestserver container startup #8463
    • Fixed filename for the configuration file in Vttestserver #8809

    CI/Build

    Build/CI

    • Update release process #7759
    • [ci] Add errcheck to golangci-lint #7996
    • gomod: do not replace GRPC #8416
    • consolidation: Fix flaky test #8417
    • Small build improvements #8418
    • proto: upgrade vtprotobuf version #8454
    • Automatically use the latest tag of Vitess for cluster upgrade E2E test #8471
    • Addition of known issues to release notes #8482
    • Fix Cluster 14 flakiness #8494
    • add question on backporting back to PR template #8541
    • Enable GitHub Actions concurrency feature #8546
    • Updated Makefile do_release script to include godoc steps #8550
    • Changing codahale/hdrhistogram import path #8637
    • Upgrade to Go 1.17 #8640
    • Fixes for reparent endtoend test flakiness #8642
    • hooks: remove govet because it already runs as part of golangci-lint #8674
    • Enhancement of the release instructions #8790
    • Remove consul-api usage in favor of official consul/api #8794
    • Enhancement of the release notes generation #8877

    Governance

    • update MAINTAINERS and CODEOWNERS for deepthi, pH14 and vmg #8675

    Query Serving

    • Cleanup: import of gh-ost test suite is complete #8459

    VReplication

    • VReplication support for non-PRIMARY KEY (Online DDL context) #8364

    Documentation

    Build/CI

    • maintainers: add Messaging as area of expertise #8578

    Cluster management

    • Enhance k8stopo flag documentation #8464
    • Update some vtctl/vtctlclient command help #8560
    • Trivial: fix the GenerateShardRanges vtctl[client] help #8586

    Examples

    • examples: update README to use new vreplication command syntax #8825

    Governance

    • update governance #8609

    Query Serving

    • vtexplain examples #8652

    VReplication

    • vtctl help/usage update #8879

    VTAdmin

    • [vtadmin] Remove outdated commands from vtadmin-web README #8496
    • Initial pass at some vtadmin docs #8526

    Enhancement

    Backup and Restore

    • Add Support for Restoring Specific Backups #8824

    Build/CI

    • Initial support for stress testing in end-to-end tests #8406
    • CI: Remove mariadb101 which is well past eol and no longer available in distributions #8446
    • Adding GitHub Self Hosted Runner tests #8721

    Cluster management

    • Vitess mixin improvements #7499
    • Expose topo_consul_lock_session_checks to allow customized consul session check #8298
    • Handle lock release with SIGHUP in VTGR #8472
    • Enhance PRS error message #8529
    • srvtopo: resilient watcher #8583
    • Add -restart_before_backup parameter for vtbackup #8608
    • srvtopo: allow unwatching from watch callbacks #8633
    • introduced cluster_operation as a new error code in vtrpc #8646
    • servenv: add --onclose_timeout flag #8651
    • Improve determinism of bootstrap #8840
    • OnlineDDL in v12: remove legacy code deprecated in earlier version #8972

    Observability

    • [vtadmin] cluster debug pages #8427
    • Export memstats for statsd backend #8777

    Query Serving

    • [tablet, queryrules] Extend query rules to check comments #8233
    • Add "show global status" support #8344
    • added batch lookup param to lookup vindex #8398
    • Online DDL/VReplication: able to read from replica #8405
    • multishard autocommit should work for bypass routing #8428
    • Add "no-scatter" flag to prohibit the use of scatter queries #8439
    • OnlineDDL: -skip-topo is always 'true' #8450
    • Add rows as keyword in sqlparser #8467
    • Periodic update of gh-ost binary #8470
    • fix when show tables record too much error log "Got unhandled packet.." #8478
    • SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8493
    • NativeDDL: analyzing added&removed unique keys #8495
    • Refactor authentication server plugin mechanism #8503
    • Update to planetscale/tengo v0.10.1-ps.v4 #8516
    • query serving to continue when topo server restarts - main #8534
    • support grpc reflection on APIs #8551
    • Customized CreateLookUpVindex to be Compatible with non-consistent Lookup Vindex #8570
    • Add additional options for configuring SSL modes as a client #8588
    • Fail plan for unsupported aggregate function #8593
    • gateway: use keyspace events when buffering requests #8700
    • Add optional query annotations, i.e. prefix SQL comments indicating the #8783
    • Slight improvement of the ACL error message #8805
    • Changed parsing for unions #8821
    • Use vt_dba user for online schema migration cutover phase #8836
    • Add parsing support for column list with subquery alias #8884
    • Add planner-version flag to vtexplain #8979
    • [Backport 12.0]: Add planner-version flag to vtexplain #9003

    TabletManager

    • Allow min TLS version for tablet to mysqld conns #8757

    VReplication

    • VReplication: Add ability to tag workflows #8388
    • Ignore SBR statements from pt-table-checksum #8396
    • Tablet Picker: add metric to record lack of available tablets #8403
    • VReplication: support different keys on source and target tables (different covered columns) #8423
    • VStream API: handle reparenting and unhealthy tablets #8445
    • Change local example to use v2 vreplication flows and change default to v2 from v1 #8553
    • Customized CreateLookUpVindex to Include a Flag to Continue Vreplication After Backfill with an Owner Provided #8572
    • Update MoveTables CLI help to reflect change we made at the beginning of #8597
    • VStreamer Field/Row Events: add Keyspace/Shard #8598
    • VStream API: Add flag to stop streaming on a reshard #8628

    VTAdmin

    • [vtadmin] cluster rpc pools #8421
    • Add support for passing custom interceptors to vtadmin grpcserver #8507

    Feature Request

    Cluster management

    • [vtctldserver] Migrate remaining ServingGraph rpcs to VtctldServer #8249
    • [grpctmclient] Add support for (bounded) per-host connection reuse #8368
    • VTGR: Vitess + MySQL group replication #8387
    • Add isActive flag to vtgr to support multi-cell topology #8780
    • [vtctld] Add SetWritable, StartReplication and StopReplication rpcs #8816
    • [vtctld] sleep/ping tablets #8826
    • [vtctld] migrate more util rpcs #8843
    • [vtctld] migrate validator rpcs #8849
    • [vtctld] localvtctldclient #8882
    • [vtctldserver] Migrate RunHealthCheck #8892
    • [vtctl] run new commands as standalone binary #8893

    VTAdmin

    • [vtadmin] [experimental] add per-api RBAC implementation #8515
    • [vtadmin] shard replication positions #8775
    • [vtadmin] GetVtctlds #8792

    Internal Cleanup

    Build/CI

    • Bump aws-sdk-go to v1.34.2 #8632
    • Upgrade consul api: go get github.com/hashicorp/consul/[email protected] #8784

    Cluster management

    • topodata: remove deprecated field 'ServedTypes' #8566
    • Upgrade to k8s 1.18 client #8762
    • [vtctl] command deprecations #8967

    Deployments

    • Remove the deprecated Helm charts and related code #8868

    Examples

    • Bump ini from 1.3.5 to 1.3.8 in /vitess-mixin/e2e #8823
    • backport operator updates to v12 branch . #9057

    General

    • Migrate k8s topo CRD to v1 api (v12 branch) #9046

    Governance

    • fix copyright #8611

    Query Serving

    • Adds flag to vttablet to disallow online DDL statements #8433
    • Remove vtrpc.LegacyErrorCode #8456
    • Allow for configuration of the minimal TLS version #8460
    • engine: allow retrying partial primitives #8727
    • srvtopo: expose WatchSrvKeyspace #8752
    • Clean up Primitive interface implementations #8901

    VReplication

    • LegacySplitCloneWorker is no longer used in any code paths #8867

    VTAdmin

    • [vtadmin] Add Options struct to wrap grpc/http options #8461
    • Update vtadmin local scripts to enable basic rbac #8801

    vtctl

    • Correctly identify backup timestamp var as a string #8891

    Other

    Examples

    • Improve the Docker local and compose examples #8685

    Performance

    Cluster management

    • throttler: don't allocate any resources unless it is actually enabled #8643

    Query Serving

    • cache: track memory usage more closely #8804

    VTAdmin

    • Use fmt.Fprintf instead of fmt.Sprintf #8922

    Testing

    Cluster management

    • Alter tests to reuse cluster components #8276
    • Adds some more orchestrator tests to vtorc #8535

    Query Serving

    • increase conn killer check to double the tx timeout value #8649
    • Added UNION testcases and auxilary code for running tests #8797
    • fixed regression in v3 for grouping by integer functions #8856

    VTAdmin

    • [vtadmin] Add a vtctld Dialer unit test #8455

    The release includes 1351 commits (excluding merges)

    Thanks to all our contributors: @Anders-PlanetScale, @GuptaManan100, @Johnny-Three, @Juneezee, @ajm188, @aquarapid, @askdba, @bnu0, @carsonoid, @choo-stripe, @dbussink, @dctrwatson, @deepthi, @demmer, @dependabot[bot], @derekperkins, @doeg, @eeSeeGee, @falun, @fatih, @frouioui, @hallaroo, @harshit-gangal, @ilikeorangutans, @mattlord, @rafael, @ritwizsinha, @rohit-nayak-ps, @sahutd, @shlomi-noach, @sonne5, @systay, @tdakkota, @tokikanno, @utk9, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.0-de06fca.tar.gz(400.35 MB)
    vitess-12.0.0-de06fca.x86_64.rpm(398.64 MB)
    vitess_12.0.0-de06fca_amd64.deb(400.36 MB)
  • v12.0.0-rc1(Oct 5, 2021)

    Release of Vitess v12.0.0

    Major Changes

    This release includes the following major changes or new features.

    Inclusive Naming

    A number of commands and RPCs have been deprecated as we move from master to primary. All functionality is backward compatible except as noted under Incompatible Changes.

    Gen4 Planner

    The newest version of the query planner, Gen4, becomes an experimental feature as part of this release. While Gen4 has been under development for a few release cycles, we have now reached parity with its predecessor, v3.

    To use Gen4, VTGate's -planner_version flag needs to be set to gen4.

    Minor Changes

    Query Serving

    • Add SHOW VITESS_REPLICATION_STATUS Query Support #8900
      • The new SHOW VITESS_REPLICATION_STATUS command/query shows the MySQL replica/replication (not vreplication) health for the vitess deployment.

    Incompatible Changes

    vtctl command output

    Wherever vtctl commands produced master or MASTER for tablet type, they now produce primary or PRIMARY. Scripts and tools that depend on parsing command output will need to change.

    Example:

    $ vtctlclient -server localhost:15999 ListAllTablets 
    zone1-0000000100 sourcekeyspace 0 primary 192.168.0.134:15100 192.168.0.134:17100 [] 2021-09-24T01:12:00Z
    zone1-0000000101 sourcekeyspace 0 rdonly 192.168.0.134:15101 192.168.0.134:17101 [] <null>
    zone1-0000000102 sourcekeyspace 0 replica 192.168.0.134:15102 192.168.0.134:17102 [] <null>
    zone1-0000000103 sourcekeyspace 0 rdonly 192.168.0.134:15103 192.168.0.134:17103 [] <null>
    

    Governance

    • Added @GuptaManan100 to maintainers #8833

    Changelog

    Bug fixes

    Build/CI

    • Update golang/x dependencies #8688
    • tests: use AtomicInt32 instead of int to fix races #8696

    Cluster management

    • Use consul lock properly #8310
    • Trivial: mysqlctl reinit_config -h was not showing help #8369
    • Port #8422 to main branch #8745
    • [vtctl] Add missing call to flag.Parse in commandGetRoutingRules #8795
    • Fix for padding in OrderAndCheckPartitions #8873

    Observability

    • schema engine: reset table size stats when a table is dropped #8634

    Query Serving

    • Fix for function calls in DEFAULT value of CREATE TABLE statement in main #8477
    • Fixing multiple issues related to onlineddl/lifecycle #8500
    • boolean values should not be parenthesised in default clause #8502
    • mysql/server: abort connection if secure transport is required #8504
    • Clear only the current shard's schemacopy rows during schema reload #8519
    • go/mysql/auth: fix error message for wrong auth method. #8525
    • Fix handshake protocol with MySQL client #8537
    • Copy parameter data from byte buffer in COM_STMT_SEND_LONG_DATA #8562
    • Negative float default #8587
    • [vtexplain] prevent panic on show tables/show full tables #8601
    • OnlineDDL: better scheduling/cancellation logic #8603
    • Fix parsing of FIRST clause in ALTER TABLE #8614
    • onlineddl Executor: build schema with DBA user #8624
    • Handle subquery merging with references correctly #8662
    • Fixing error on deletes from owning table for a lookup being populated by CreateLookupVindex after a SwitchWrites #8701
    • Fixing a panic in vtgate with OLAP mode #8722
    • default to primary tablet if not set in VStream api #8755
    • Fix bug in parsing of version comments #8770
    • Proper merge of the SysTableTableName fields when joining two routes #8771
    • fix parsing of MySQL Server Version flag #8791
    • Fix typo in pool config syntax. Fixes #8819 #8820
    • VtGate schema tracker to use provided user #8857
    • fix merging dba query as a subquery both in v3 and gen4 #8871

    VReplication

    • MoveTables: don't create unnecessary streams on the target for non-intersecting sources and targets #8090
    • VDiff: Add BIT datatype to list of byte comparable types #8401
    • Fix vreplication error metric #8483
    • Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
    • Fix VReplication logging to file and db #8521
    • Fix passing the wrong cell/cells variable to CreateLookupVindex #8590
    • Refresh SrvVSchema after an ExternalizeVindex: was missing #8670
    • Fixing missing argument in function call #8692
    • Fix how we identify MySQL generated columns #8763

    VTAdmin

    • [vtadmin-web] Set Backup status indicators to the correct colours #8410
    • [vtadmin] Fix bad copy-paste in pool config flag parsing #8518

    vtctl

    • Print actual current datetime for vtctl dry run commands #8778

    vttestserver

    • Remove lingering mysqld sock files on vttestserver container startup #8463
    • Fixed filename for the configuration file in Vttestserver #8809

    CI/Build

    Build/CI

    • Update release process #7759
    • [ci] Add errcheck to golangci-lint #7996
    • gomod: do not replace GRPC #8416
    • consolidation: Fix flaky test #8417
    • Small build improvements #8418
    • proto: upgrade vtprotobuf version #8454
    • Automatically use the latest tag of Vitess for cluster upgrade E2E test #8471
    • Addition of known issues to release notes #8482
    • Fix Cluster 14 flakiness #8494
    • add question on backporting back to PR template #8541
    • Enable GitHub Actions concurrency feature #8546
    • Updated Makefile do_release script to include godoc steps #8550
    • Changing codahale/hdrhistogram import path #8637
    • Upgrade to Go 1.17 #8640
    • Fixes for reparent endtoend test flakiness #8642
    • hooks: remove govet because it already runs as part of golangci-lint #8674
    • Enhancement of the release instructions #8790
    • Remove consul-api usage in favor of official consul/api #8794
    • Enhancement of the release notes generation #8877

    Governance

    • update MAINTAINERS and CODEOWNERS for deepthi, pH14 and vmg #8675

    Query Serving

    • Cleanup: import of gh-ost test suite is complete #8459

    VReplication

    • VReplication support for non-PRIMARY KEY (Online DDL context) #8364

    Documentation

    Build/CI

    • maintainers: add Messaging as area of expertise #8578

    Cluster management

    • Enhance k8stopo flag documentation #8464
    • Update some vtctl/vtctlclient command help #8560
    • Trivial: fix the GenerateShardRanges vtctl[client] help #8586

    Examples

    • examples: update README to use new vreplication command syntax #8825

    Governance

    • update governance #8609

    Query Serving

    • vtexplain examples #8652

    VReplication

    • vtctl help/usage update #8879

    VTAdmin

    • [vtadmin] Remove outdated commands from vtadmin-web README #8496
    • Initial pass at some vtadmin docs #8526

    Enhancement

    Backup and Restore

    • Add Support for Restoring Specific Backups #8824

    Build/CI

    • Initial support for stress testing in end-to-end tests #8406
    • CI: Remove mariadb101 which is well past eol and no longer available in distributions #8446
    • Adding GitHub Self Hosted Runner tests #8721

    Cluster management

    • Vitess mixin improvements #7499
    • Expose topo_consul_lock_session_checks to allow customized consul session check #8298
    • Handle lock release with SIGHUP in VTGR #8472
    • Enhance PRS error message #8529
    • srvtopo: resilient watcher #8583
    • Add -restart_before_backup parameter for vtbackup #8608
    • srvtopo: allow unwatching from watch callbacks #8633
    • introduced cluster_operation as a new error code in vtrpc #8646
    • servenv: add --onclose_timeout flag #8651
    • Improve determinism of bootstrap #8840

    Observability

    • [vtadmin] cluster debug pages #8427
    • Export memstats for statsd backend #8777

    Query Serving

    • [tablet, queryrules] Extend query rules to check comments #8233
    • Add "show global status" support #8344
    • added batch lookup param to lookup vindex #8398
    • Online DDL/VReplication: able to read from replica #8405
    • multishard autocommit should work for bypass routing #8428
    • Add "no-scatter" flag to prohibit the use of scatter queries #8439
    • OnlineDDL: -skip-topo is always 'true' #8450
    • Add rows as keyword in sqlparser #8467
    • Periodic update of gh-ost binary #8470
    • fix when show tables record too much error log "Got unhandled packet.." #8478
    • SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8493
    • NativeDDL: analyzing added&removed unique keys #8495
    • Refactor authentication server plugin mechanism #8503
    • Update to planetscale/tengo v0.10.1-ps.v4 #8516
    • query serving to continue when topo server restarts - main #8534
    • support grpc reflection on APIs #8551
    • Customized CreateLookUpVindex to be Compatible with non-consistent Lookup Vindex #8570
    • Add additional options for configuring SSL modes as a client #8588
    • Fail plan for unsupported aggregate function #8593
    • gateway: use keyspace events when buffering requests #8700
    • Add optional query annotations, i.e. prefix SQL comments indicating the #8783
    • Slight improvement of the ACL error message #8805
    • Changed parsing for unions #8821
    • Use vt_dba user for online schema migration cutover phase #8836
    • Add parsing support for column list with subquery alias #8884

    TabletManager

    • Allow min TLS version for tablet to mysqld conns #8757

    VReplication

    • VReplication: Add ability to tag workflows #8388
    • Ignore SBR statements from pt-table-checksum #8396
    • Tablet Picker: add metric to record lack of available tablets #8403
    • VReplication: support different keys on source and target tables (different covered columns) #8423
    • VStream API: handle reparenting and unhealthy tablets #8445
    • Change local example to use v2 vreplication flows and change default to v2 from v1 #8553
    • Customized CreateLookUpVindex to Include a Flag to Continue Vreplication After Backfill with an Owner Provided #8572
    • Update MoveTables CLI help to reflect change we made at the beginning of #8597
    • VStreamer Field/Row Events: add Keyspace/Shard #8598
    • VStream API: Add flag to stop streaming on a reshard #8628

    VTAdmin

    • [vtadmin] cluster rpc pools #8421
    • Add support for passing custom interceptors to vtadmin grpcserver #8507

    Feature Request

    Cluster management

    • [vtctldserver] Migrate remaining ServingGraph rpcs to VtctldServer #8249
    • [grpctmclient] Add support for (bounded) per-host connection reuse #8368
    • VTGR: Vitess + MySQL group replication #8387
    • Add isActive flag to vtgr to support multi-cell topology #8780
    • [vtctld] Add SetWritable, StartReplication and StopReplication rpcs #8816
    • [vtctld] sleep/ping tablets #8826
    • [vtctld] migrate more util rpcs #8843
    • [vtctld] migrate validator rpcs #8849
    • [vtctld] localvtctldclient #8882
    • [vtctldserver] Migrate RunHealthCheck #8892
    • [vtctl] run new commands as standalone binary #8893

    VTAdmin

    • [vtadmin] [experimental] add per-api RBAC implementation #8515
    • [vtadmin] shard replication positions #8775
    • [vtadmin] GetVtctlds #8792

    Internal Cleanup

    Build/CI

    • Bump aws-sdk-go to v1.34.2 #8632
    • Upgrade consul api: go get github.com/hashicorp/consul/[email protected] #8784

    Cluster management

    • topodata: remove deprecated field 'ServedTypes' #8566
    • Upgrade to k8s 1.18 client #8762

    Deployments

    • Remove the deprecated Helm charts and related code #8868

    Examples

    • Bump ini from 1.3.5 to 1.3.8 in /vitess-mixin/e2e #8823

    Governance

    • fix copyright #8611

    Query Serving

    • Adds flag to vttablet to disallow online DDL statements #8433
    • Remove vtrpc.LegacyErrorCode #8456
    • Allow for configuration of the minimal TLS version #8460
    • engine: allow retrying partial primitives #8727
    • srvtopo: expose WatchSrvKeyspace #8752
    • Clean up Primitive interface implementations #8901

    VReplication

    • LegacySplitCloneWorker is no longer used in any code paths #8867

    VTAdmin

    • [vtadmin] Add Options struct to wrap grpc/http options #8461
    • Update vtadmin local scripts to enable basic rbac #8801

    vtctl

    • Correctly identify backup timestamp var as a string #8891

    Other

    Examples

    • Improve the Docker local and compose examples #8685

    Performance

    Cluster management

    • throttler: don't allocate any resources unless it is actually enabled #8643

    Query Serving

    • cache: track memory usage more closely #8804

    VTAdmin

    • Use fmt.Fprintf instead of fmt.Sprintf #8922

    Testing

    Cluster management

    • Alter tests to reuse cluster components #8276
    • Adds some more orchestrator tests to vtorc #8535

    Query Serving

    • increase conn killer check to double the tx timeout value #8649
    • Added UNION testcases and auxilary code for running tests #8797
    • fixed regression in v3 for grouping by integer functions #8856

    VTAdmin

    • [vtadmin] Add a vtctld Dialer unit test #8455

    The release includes 1281 commits (excluding merges)

    Thanks to all our contributors: @Anders-PlanetScale, @GuptaManan100, @Johnny-Three, @Juneezee, @ajm188, @aquarapid, @askdba, @bnu0, @choo-stripe, @dbussink, @dctrwatson, @deepthi, @demmer, @dependabot[bot], @derekperkins, @doeg, @eeSeeGee, @falun, @fatih, @frouioui, @hallaroo, @harshit-gangal, @ilikeorangutans, @mattlord, @rafael, @ritwizsinha, @rohit-nayak-ps, @sahutd, @shlomi-noach, @sonne5, @systay, @tdakkota, @tokikanno, @utk9, @vmg

    Source code(tar.gz)
    Source code(zip)
    vitess-12.0.0-rc1-bd0a4ed.tar.gz(401.79 MB)
    vitess-12.0.0_rc1-bd0a4ed.x86_64.rpm(400.06 MB)
    vitess_12.0.0-rc1-bd0a4ed_amd64.deb(401.79 MB)
  • v11.0.1(Sep 9, 2021)

    Bug fixes

    Cluster management

    • Fix problematic ERS cases #8744

    Query Serving

    • Handle subquery merging with references correctly #8661
    • onlineddl Executor: build schema with DBA user #8667
    • Backport to 11: Fixing a panic in vtgate with OLAP mode #8746
    • Backport into 11: default to primary tablet if not set in VStream api #8766

    VReplication

    • Refresh SrvVSchema after an ExternalizeVindex: was missing #8669

    CI/Build

    Build/CI

    • Vitess Release 11.0.0 #8549
    • Backport to 11: Updated Makefile do_release script to include godoc steps #8787

    The release includes 18 commits (excluding merges) Thanks to all our contributors: @aquarapid, @askdba, @frouioui, @harshit-gangal, @rohit-nayak-ps, @shlomi-noach, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.1-92ac1ff.tar.gz(397.08 MB)
    vitess-11.0.1-92ac1ff.x86_64.rpm(394.80 MB)
    vitess_11.0.1-92ac1ff_amd64.deb(397.09 MB)
  • v11.0.0(Jul 27, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Bug fixes

    Build/CI

    • update moby/term to fix darwin build issue #7787
    • Removing Linux/amd64 specific subdependencies #7796
    • CI: run some tests inside docker to workaround GH Actions issue #7868
    • Fix flaky race condition in vtexplain #7930
    • Fixing flaky endtoend/tablegc tests #7947
    • Add libwww-perl to allow pt-osc wrapper to work in vitess/lite #8141

    Cluster management

    • Add missing return on a failed GetShard call in FindAllShardsInKeyspace #7992
    • Increase the default srv_topo_timeout #8011
    • parser: support index name in FOREIGN KEY clause; Online DDL to reject FK clauses #8058
    • Correctly parse cell names with dashes in tablet aliases #8167
    • /healthz should not report ok when vttablet is not connected to mysql #8238
    • [topo] Refactor ExpandCells to not error on valid aliases #8291
    • tm state: don't populate metadata in updateLocked #8362
    • [grpcvtctldserver] Fix backup detail limit math #8402

    Query Serving

    • fix select star, col1 orderby col1 bug. #7743
    • VTExplain: Add support for multi table join query #7829
    • Routing of Information Schema Queries #7841
    • Fix + integration test for keyspaces_to_watch routing regression [fixes #7882] #7873
    • Revert "[tablet, queryrules] Extend query rules to check comments" #7897
    • vtctl: return error on invalid ddl_strategy #7923
    • Panic when EOF after @ symbol #7925
    • Fixes encoding of sql strings #8033
    • Make TwoPC engine initialization async. #8048
    • Fix for issue with information_schema queries with both table name and schema name predicates #8087
    • Fix for transactions not allowed to finish during PlannedReparentShard #8089
    • Set fully parsed to false when ignoring errors #8094
    • Fix for query and sub query with limits #8097
    • Fix buffering when using reserved connections #8102
    • Parse generated columns in DDL #8117
    • More explicit message on online DDL parsing error #8118
    • healthcheck: attempt to update primary only if the current tablet is serving #8121
    • PRIMARY in index hint list for master #8160
    • Signed int parse #8189
    • Delete table reference alias support #8393
    • Fix for function calls in DEFAULT value of CREATE TABLE statement in release-11.0 #8476
    • Backport: Fixing multiple issues related to onlineddl/lifecycle #8517
    • boolean values should not be parenthesised in default clause - release 11 #8531

    VReplication

    • VDiff: Use byte compare if weight_string() returns null for either source or target #7696
    • Rowlog: Update rowlog for the API change made for the vstream skew alignment feature #7809
    • Pad binlog values for binary() columns to match the value returned by mysql selects #7969
    • Change vreplication error metric name to start with a string to pass prometheus validations #7983
    • Pass the provided keyspace through to UpdateDisableQueryService rather than hard-coding the sourceKeyspace #8020
    • Fix vreplication timing metrics #8024
    • Switchwrites: error if no tablets available on target for reverse replication #8142
    • Remove noisy vexec logs #8144
    • VReplicationExec: don't create new stats objects on a select #8166
    • Binlog JSON Parser: handle inline types correctly for large documents #8187
    • Schema Tracking Flaky Test: Ignore unrelated gtid progress events #8283
    • Adds padding to keyrange comparison #8296
    • VReplication Reverse Workflows: add keyspace scope to vindex while creating reverse vreplication streams #8385
    • OnlineDDL/Vreplication stress test: investigating failures #8390
    • Ignore SBR statements from pt-table-checksum #8396
    • Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
    • VDiff: Add BIT datatype to list of byte comparable types #8401
    • Fix excessive VReplication logging to file and db #8521

    VTAdmin

    • Add missing return in vtctld-* DSN case, and log any flag that gets ignored #7872
    • [vtadmin-web] Do not parse numbers/booleans in URL query parameters by default #8100
    • [vtadmin-web] Small bugfix where stream source displayed instead of target #8311

    CI/Build

    Build/CI

    • Planbuilder: Fix fuzzer #7952
    • java: Bump SNAPSHOT version to 11.0.0-SNAPSHOT after Vitess release v10 #7968
    • Add optional TLS feature to gRPC servers #8049
    • Add image and base image arguments to build.sh #8064
    • [fuzzing] Add report #8128
    • CI: fail PR if it does not have required labels #8147
    • trigger pr-labels workflow when labels added/removed from PR #8157
    • Add vmg as a maintainer #8186
    • Refactor vtorc endtoend tests #8215
    • Use shared testing helper which guards against races in tmclient construction #8300
    • moved multiple vtgate and tabletgateway to individual shards #8305
    • include squashed PRs in release notes #8326
    • pr-labels workflow: only check actual PRs #8327
    • Heuristic fix for a flaky test: allow for some noise to pass through #8339
    • ci: upgrade to Go 1.16 #8274
    • mod: upgrade etcd to stable version #8357
    • Resolve go.mod issue with direct dependency on planetscale/tengo #8383
    • Initial GitHub Docker Build Setup #8399

    Cluster management

    • make: build vitess as static binaries by default #7795
    • Fix TableGC flaky test by reducing check interval #8270

    Java

    • Bump commons-io from 2.6 to 2.7 in /java #7953

    Other

    • Bump MySQL version for docker builds to 8.0.23 #7811

    Query Serving

    • Online DDL Vreplication test suite: adding tests #8213
    • Online DDL/VReplication: more passing tests (no UK/PK) #8225
    • Online DDL: reject ALTER TABLE...RENAME statements #8227
    • Online DDL/VReplication: fail on existence of FOREIGN KEY #8228
    • Online DDL/VReplication: test PK column case change #8336
    • Online DDL/VReplication: add PK DATETIME->TIMESTAMP test #8338
    • Bugfix: assign cexpr.references for column CONVERTed to utf8mb4 #8355

    VReplication

    • Towards a VReplication/OnlineDDL testing suite #8181
    • Online DDL/Vreplication: column type awareness #8239
    • remove duplicate ReadMigrations, fixing build #8258
    • VReplication (and by product, Online DDL): support GENERATED column as part of PRIMARY KEY #8335

    vttestserver

    • docker/vttestserver: Set max_connections in default-fast.cnf at container build time #7810

    Documentation

    Cluster management

    • Enhance k8stopo flag documentation #8458

    Build/CI

    • Update version for latest snapshot #7801
    • v10 GA Release Notes #7964
    • Updates and Corrections to v9 Release Notes #8295

    Query Serving

    • Update link to 'Why FK not supported in Online DDL' blog post #8372

    Other

    • correct several wrong words #7822
    • MAINTAINERS.md: Update enisoc's email. #7909
    • Modification of the Markdown list format in the release notes template #7962
    • Update vreplicator docs #8014
    • Release notes: add Known Issues #8027

    Enhancement

    Observability

    • [trace] Add logging support to opentracing plugins #8289

    Build/CI

    • Makefile: add cross-build target for cross-compiling client binaries #7806
    • git: improve signoff detection #7852
    • Release notes generation #7932
    • vitess/lite docker build fails for mysql80 #7943
    • Fix a gofmt warning #7959
    • docker/vttestserver/run.sh: Add $CHARSET environment variable #7970
    • docker/lite/install_dependencies.sh: If the dependency install loop reaches its max number of retries, consider that a failure; exit the script nonzero so the build halts. #7976
    • Add commit count and authors to the release notes #7982
    • Automate version naming and release tagging #8034
    • Make sure to only allow codegen when the rest of the code compiles #8169
    • Add options to vttestserver to pass -foreign_key_mode, -enable_online_ddl, and -enable_direct_ddl through to vtcombo #8177

    Cluster management

    • Dynamic throttle metric threshold #7742
    • Bump Bootstrap version, per CVE-2018-14040 (orchestrator/vtorc) #7824
    • Check tablet alias before removing after error stream #7915
    • vttablet/tabletmanager - add additional test for tmstate.Open() #7993
    • Add vttablet_restore_done hook #8007
    • Added ValidateVSchema #8012
    • Add RemoteOperationTimeout to both legacy and grpc ChangeTabletType implementations. #8052
    • [vtctldserver] Add guard against self-reparent, plus misc updates #8084
    • [tm_state] updateLocked should re-populate local metadata tables to reflect promotion rule changes #8107
    • [vtctld] Migrate ApplyVSchema to VtctldServer #8113
    • vtctl.generateShardRanges -> key.GenerateShardRanges #8134
    • etcd: add grpc.WithBlock to client config #8205
    • [workflow] Add tracing to GetWorkflows endpoint #8266
    • [vtctldclient] Add legacy shim #8284
    • [vtctldserver] Add tracing #8285
    • [vtctldserver] Add additional backup info fields #8321
    • [workflow] Call scanWorkflow concurrently #8272

    Query Serving

    • [Gen4] Implemented table alias functionality in the semantic analysis #7629
    • Improved error messages (tabletserver) #7747
    • Allow modification of tablet unhealthy_threshold via debugEnv #7753
    • [Gen4] Initial Horizon Planning #7775
    • [tablet, queryrules] Extend query rules to check comments #7784
    • Add support for showing global gtid executed per shard #7856
    • Minor cleanups around errors on the vtgate #7864
    • log unsupported queries #7865
    • Show databases like #7912
    • Add rank as reserved keyword #7944
    • Online DDL: introducing ddl_strategy -singleton-context flag #7946
    • Ignore the error and log as warn if not able to validate the current system setting for check and ignore case #8004
    • Detect and signal schema changes on vttablets #8005
    • DDL bypass plan #8013
    • Online DDL: progress & ETA for Vreplication migrations #8015
    • Scatter errors as warning in olap query #8018
    • livequeryz: livequeryz/terminate link should be relative #8025
    • Handle online DDL user creation if we do not have SUPER privs (e.g. AWS Aurora) #8038
    • Update the schema copy with minimal changes #8067
    • Gen4: Support for order by column not present in projection #8070
    • Schema tracking in vtgate #8074
    • protobuf: upgrade #8075
    • added mysql 8.0 reserved keywords #8086
    • Fix ghost/pt-osc in the external DB case where MySQL might be reporting #8115
    • Support for vtgate -enable_direct_ddl flag #8116
    • add transaction ID to query log for acquisition and analysis #8133
    • Inline reference #8136
    • Improve ScatterErrorsAsWarnings functionality #8139
    • Gen4: planning Select IN #8155
    • Primary key name #8188
    • Online DDL: read and publish gh-ost FATAL message where possible #8192
    • Expose inner net.Conn to be able to write better unit tests #8217
    • vtgate: validate important flags during startup #8218
    • Online DDL/VReplication: AUTO_INCREMENT support and tests #8223
    • [Gen4] some renaming from v4 to Gen4 #8234
    • Schema Tracking: Optimize and Bug Fix #8243
    • gen4: minor refactoring to improve readability #8245
    • gen4: plan more opcodes #8254
    • Online DDL: report rows_copied for migration (initial support in gh-ost) #8255
    • Schema tracking: new tables in sharded keyspace #8256
    • [parser] use table_alias for ENGINE option in CREATE TABLE stmt #8307
    • Add /debug/env for vtgate #8292
    • gen4: outer joins #8312
    • Gen4: expand star in projection list #8325
    • gen4: Fail all queries not handled well by gen4 #8359
    • Gen4 fail more2 #8382
    • SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8532
    • [11.0] query serving to continue when topo server restarts #8533
    • [11.0] Disable allowing set statements on system settings by default #8540

    VReplication

    • Change local example to use v2 vreplication flows and make v2 flows as the default #8527
    • Use Dba user when Vexec is runAsAdmin #7731
    • Add table for logging stream errors, state changes and key workflow steps #7831
    • Fix some static check warning #7960
    • VSchema Validation on ReshardWorkflow Creation #7977
    • Tracking rows_copied #7980
    • Vdiff formatting improvements #8079
    • Ignore generated columns in workflows #8129
    • VReplication Copy Phase: Increase default replica lag tolerance. Also make it and copy timeout modifiable via flags #8130
    • Materialize: Add additional comparison operators in Materialize and fix bug where they not applied for sharded keyspaces #8247
    • Copy Phase: turn on OptimizeInserts by default #8248
    • Tracker/VStreamer: only reload schema for tables in current database and not for internal table artifacts #8257
    • Online DDL/Vreplication suite: support ENUM->VARCHAR/TEXT type change #8275
    • Added TableName to DiffReport struct. #8279
    • New VReplication lag metric #8306
    • VStream API: add heartbeat for idle streams #8244
    • Online DDL/VReplication: support non-UTF8 character sets #8322
    • Online DDL/Vreplication suite: fix test for no shared UK #8334
    • Online DDL/VReplication: support DROP+ADD column of same name #8337
    • Online DDL/VReplication test suite: support ENUM as part of PRIMARY KEY #8345
    • Change local example to use v2 vreplication flows #8527
    • Tablet Picker: add metric to record lack of available tablets #8403

    VTAdmin

    • [vtadmin-web] Add useSyncedURLParam hook to persist filter parameter in the URL #7857
    • [vtadmin-web] Display more data on /gates view and add filtering #7876
    • [vtadmin] Promote ErrNoSchema to a TypedError which returns http 404 #7885
    • [vtadmin] gate/tablet/vtctld FQDNs #7886
    • [vtadmin-web] Display vindex data on Schema view #7917
    • [vtadmin-web] Add filtering and source/target shards to Workflows view #7948
    • [vtadmin-web] Add filtering + shard counts/status to Keyspaces view #7991
    • [vtadmin-web] Display shard state on Tablets view + extract tablet utilities #7999
    • [vtadmin] experimental tabletdebug #8003
    • [vtadmin-web] Display timestamps + stream counts on Workflows view #8009
    • [vtadmin-web] Updates to table styling + other CSS #8072
    • [vtadmin-web] Add Tab components #8119
    • [vtadmin-api] Update GetTablet to use alias instead of hostname #8163
    • [vtadmin-api] Rename flag 'http-tablet-fqdn-tmpl' to 'http-tablet-url-tmpl' + update vtadmin flags for local example #8164
    • Add --tracer flag to vtadmin and actually start tracing #8165
    • [vtadmin-web] The hastiest Tablet view (+ experimental debug vars) #8170
    • [vtadmin-web] Add tabs to Workflow view #8203
    • [vtctld] Add GetSrvVSchemas command #8221
    • [vtadmin-api] Add HTTP endpoints for /api/srvvschemas and /api/srvvschema/{cluster}/{cell} #8226
    • [vtadmin-web] Add QPS and VReplicationQPS charts to Tablet view #8263
    • [vtadmin-web] Add client-side error handling interface + Bugsnag implementation #8287
    • [vtadmin-web] Add chart for stream vreplication lag across all streams in a workflow #8331
    • [vtctldproxy] Add more annotations to vtctld Dial calls #8346

    vttestserver

    • Vttest create db #7989
    • Adds an environment variable to set the MySQL max connections limit in vttestserver docker image #8210

    Feature Request

    Build/CI

    • [trace] Add optional flags to support flexible jaeger configs #8199

    Cluster management

    • Add VtctldServer to vtcombo #7896
    • [vtctldserver] Migrate routing rules RPCs, and also RebuildVSchemaGraph #8197
    • [vtctldserver] Migrate CellInfo, CellAlias rw RPCs #8219
    • [vtctldserver] Add RefreshState RPCs #8232

    Query Serving

    • VTGate grpc implementation of Prepare and CloseSession #8211
    • Schema tracking: One schema load at a time per keyspace #8224

    VTAdmin

    • [vtadmin-web] Add DataFilter + Workspace layout components #8032
    • [vtadmin-web] Add Tooltip + HelpTooltip components #8076
    • [vtadmin-web] Add initial Stream view, render streams on Workflow view #8091
    • [vtadmin-web] The hastiest-ever VTExplain UI #8092
    • [vtadmin-web] Add source-map-explorer util #8093
    • [vtadmin-web] Add Keyspace detail view #8111
    • [vtadmin-api] Add GetKeyspace endpoint #8125
    • [workflow] Add vreplication_log data to workflow protos, and VtctldServer.GetWorkflows method #8261
    • [vtadmin] Add debug endpoints #8268

    Internal Cleanup

    Build/CI

    • Makefile: fix cross-build comments #8246
    • remove unused hooks with refs to master branch #8250
    • [vtctldserver] Update tests to prevent races during tmclient init #8320
    • sks-keyservers.net seems to be finally dead, replace with #8363
    • endtoend: change log level of etcd start from error to info #8370

    Cluster management

    • Online DDL: code cleanup #7589
    • [wrangler|topotools] Migrate UpdateShardRecords, RefreshTabletsByShard, and {Get,Save}RoutingRules #7965
    • Tear down old stream_migrater shim, now that we're fully in package workflow #8073
    • [mysqlctl] Restructure MetadataManager to reduce public API surface area #8152
    • naming: master to primary #8251
    • vtorc: code cleanup #8269

    Query Serving

    • Plan StreamExecute Queries #7941

    VReplication

    • [workflow] extract migration targets from wrangler #7934
    • [wrangler|workflow] Extract vrStream type to workflow.VReplicationStream #7966
    • [wrangler|workflow] Extract workflowState and workflowType out to package workflow #7967
    • [wrangler|workflow] extract *wrangler.streamMigrater to workflow.StreamMigrator #8008
    • [workflow] Migrate getCellsWith{Shard,Table}ReadsSwitched, TrafficSwitchDirection and TableRemovalType to package workflow #8190
    • [workflow] Cleanup wrangler wrappers, migrate checkIfJournalExistsOnTablet to package workflow #8193
    • Backports of #8403 #8483 #8489 #8401 #8521 #8396 from main into release 11.0 #8536

    VTAdmin

    • [vtadmin-api] Replace magic numbers with net/http constants #8127
    • [vtadmin-web] Move single-entity view components into subfolders #8202
    • [vtadmin] Ensure we log any errors when closing the tracer #8262

    Other

    Build/CI

    • add vtctldclient to binary directory #7889

    Cluster management

    • vttablet/tabletmanager: add isInSRVKeyspace/isShardServing #7929

    Other

    • Change VitessInputFormat key type #199
    • Online DDL plan via Send; "singleton" migrations on tablets #7785
    • flaky onlineddl tests: reduce -online_ddl_check_interval #7847
    • Looking into flaky endtoend upgrade test #7900
    • fixing flaky upgrade test #7901

    Query Serving

    • Introduce Concatenated Fixed-width Composite or CFC vindex #7537
    • Add common tags for stats backends that support it #7651
    • Add support for showing global vgtid executed #7797
    • perf: vttablet/mysql optimizations #7800
    • Fix bug with reserved connections to stale tablets #7879
    • Memory Sort to close the goroutines when callback returns error #7903
    • OnlineDDL: more migration check ticks upon migration start #7961
    • Update gh-ost binary to v1.1.3 #8021
    • VReplication Online DDL: fix classification of virtual columns #8043

    VReplication

    • VReplicationErrors metric: use . as delimiter instead of _ to behave well with Prometheus #7807

    VTAdmin

    • Update GetSchema filtering to exclude shards where IsMasterServing but no MasterAlias #7805
    • [vtadmin-api] Reintroduce include_non_serving_shards opt to GetSchema #7814
    • [vtadmin-web] Add DataCell component #7817
    • Rewrite useTableDefinitions hook as getTableDefinitions util #7821
    • [vtadmin-web] Display (approximate) table sizes + row count on /schemas view #7826
    • [vtadmin-web] Add Pip + TabletServingPip components #7827

    Performance

    Query Serving

    • vttablet: stream consolidation #7752
    • perf: optimize bind var generation #7828

    VReplication

    • Optimize the catchup phases by filtering out rows which not within range of copied pks during inserts #7708
    • Ability to compress gtid when stored in _vt.vreplication's pos column #7877
    • Performance & benchmarks (table copying) #7881
    • Dynamic packet sizing #7933
    • perf: vreplication client CPU usage #7951
    • VDIff: fix performance regression introduced by progress logging #8016
    • proto: Generate faster code using vtprotobuf #8173
    • proto: enable pooling for vreplication #8273

    Testing

    Build/CI

    • Attempt to fix TLS Server Flaky test #7842
    • TestMakeCommonTags Flaky Test: match elements #7843
    • upgrade tests: test against v9.0.0 #7848
    • FOSSA scan added #7862
    • ci: fix all racy tests #7904
    • CI: Revert docker change for unit tests from #7868 #7940
    • Add online ddl on start queries to schema list in vtexplain tablet #8397

    Java

    • [Java] JDBC mysql driver test #8154

    Other

    • Fuzzing: Fixup oss-fuzz build script #7782
    • Wrangler tests: Return a fake tablet in the wrangler test dialer to avoid tablet picker errors spamming the test logs #7863

    Query Serving

    • clean up test #7816
    • tabletserver: fix flaky test #7851
    • Planbuilder: Add fuzzer #7902
    • mysql: Small adjustments to fuzzer #7907
    • vtgate/engine: Add fuzzer #7914
    • Adding Fuzzer Test Cases #8106
    • Addition of fuzzer issues #8195

    VReplication

    • vstreamer: Add fuzzer #7918

    vttestserver

    • Speedup new vttestserver tests #8229
    • Vttestserver docker test #8253

    Cluster management

    • Make timestamp authoritative for master information #8381

    The release includes 1080 commits (excluding merges)

    Thanks to all our contributors: @AdamKorcz, @GuptaManan100, @Hellcatlk, @Johnny-Three, @acharisshopify, @ajm188, @alexrs, @aquarapid, @askdba, @deepthi, @doeg, @dyv, @enisoc, @frouioui, @gedgar, @guidoiaquinti, @harshit-gangal, @hkdsun, @idvoretskyi, @jmoldow, @kirs, @mcronce, @narcsfz, @noxiouz, @rafael, @rohit-nayak-ps, @setassociative, @shlomi-noach, @systay, @tokikanno, @vmg, @wangmeng99, @yangxuanjia, @zhangshj-inspur

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.0-aa798b8.tar.gz(397.07 MB)
    vitess-11.0.0-aa798b8.x86_64.rpm(394.82 MB)
    vitess_11.0.0-aa798b8_amd64.deb(397.08 MB)
  • v11.0.0-rc1(Jul 6, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Bug fixes

    Build/CI

    • update moby/term to fix darwin build issue #7787
    • Removing Linux/amd64 specific subdependencies #7796
    • CI: run some tests inside docker to workaround GH Actions issue #7868
    • Fix flaky race condition in vtexplain #7930
    • Fixing flaky endtoend/tablegc tests #7947
    • Add libwww-perl to allow pt-osc wrapper to work in vitess/lite #8141

    Cluster management

    • Add missing return on a failed GetShard call in FindAllShardsInKeyspace #7992
    • Increase the default srv_topo_timeout #8011
    • parser: support index name in FOREIGN KEY clause; Online DDL to reject FK clauses #8058
    • Correctly parse cell names with dashes in tablet aliases #8167
    • /healthz should not report ok when vttablet is not connected to mysql #8238
    • [topo] Refactor ExpandCells to not error on valid aliases #8291
    • tm state: don't populate metadata in updateLocked #8362
    • [grpcvtctldserver] Fix backup detail limit math #8402

    Query Serving

    • fix select star, col1 orderby col1 bug. #7743
    • VTExplain: Add support for multi table join query #7829
    • Routing of Information Schema Queries #7841
    • Fix + integration test for keyspaces_to_watch routing regression [fixes #7882] #7873
    • Revert "[tablet, queryrules] Extend query rules to check comments" #7897
    • vtctl: return error on invalid ddl_strategy #7923
    • Panic when EOF after @ symbol #7925
    • Fixes encoding of sql strings #8033
    • Make TwoPC engine initialization async. #8048
    • Fix for issue with information_schema queries with both table name and schema name predicates #8087
    • Fix for transactions not allowed to finish during PlannedReparentShard #8089
    • Set fully parsed to false when ignoring errors #8094
    • Fix for query and sub query with limits #8097
    • Fix buffering when using reserved connections #8102
    • Parse generated columns in DDL #8117
    • More explicit message on online DDL parsing error #8118
    • healthcheck: attempt to update primary only if the current tablet is serving #8121
    • PRIMARY in index hint list for master #8160
    • Signed int parse #8189
    • Delete table reference alias support #8393

    VReplication

    • VDiff: Use byte compare if weight_string() returns null for either source or target #7696
    • Rowlog: Update rowlog for the API change made for the vstream skew alignment feature #7809
    • VReplication: Pad binlog values for binary() columns to match the value returned by mysql selects #7969
    • Change vreplication error metric name to start with a string to pass prometheus validations #7983
    • Pass the provided keyspace through to UpdateDisableQueryService rather than hard-coding the sourceKeyspace #8020
    • vreplication: fix vreplication timing metrics #8024
    • Switchwrites: error if no tablets available on target for reverse replication #8142
    • VReplication: remove noisy vexec logs #8144
    • VReplicationExec: don't create new stats objects on a select #8166
    • Binlog JSON Parser: handle inline types correctly for large documents #8187
    • Schema Tracking Flaky Test: Ignore unrelated gtid progress events #8283
    • Adds padding to keyrange comparison #8296
    • VReplication Reverse Workflows: add keyspace scope to vindex while creating reverse vreplication streams #8385

    VTAdmin

    • Add missing return in vtctld-* DSN case, and log any flag that gets ignored #7872
    • [vtadmin-web] Do not parse numbers/booleans in URL query parameters by default #8100
    • [vtadmin-web] Small bugfix where stream source displayed instead of target #8311

    CI/Build

    Build/CI

    • Planbuilder: Fix fuzzer #7952
    • java: Bump SNAPSHOT version to 11.0.0-SNAPSHOT after Vitess release v10 #7968
    • Add optional TLS feature to gRPC servers #8049
    • Add image and base image arguments to build.sh #8064
    • [fuzzing] Add report #8128
    • CI: fail PR if it does not have required labels #8147
    • trigger pr-labels workflow when labels added/removed from PR #8157
    • Add vmg as a maintainer #8186
    • Refactor vtorc endtoend tests #8215
    • Use shared testing helper which guards against races in tmclient construction #8300
    • moved multiple vtgate and tabletgateway to individual shards #8305
    • include squashed PRs in release notes #8326
    • pr-labels workflow: only check actual PRs #8327
    • Heuristic fix for a flaky test: allow for some noise to pass through #8339
    • ci: upgrade to Go 1.16 #8274
    • mod: upgrade etcd to stable version #8357
    • Resolve go.mod issue with direct dependency on planetscale/tengo #8383
    • Initial GitHub Docker Build Setup #8399

    Cluster management

    • make: build vitess as static binaries by default #7795
    • Fix TableGC flaky test by reducing check interval #8270

    Java

    • Bump commons-io from 2.6 to 2.7 in /java #7953

    Other

    • Bump MySQL version for docker builds to 8.0.23 #7811

    Query Serving

    • Online DDL Vreplication test suite: adding tests #8213
    • Online DDL/VReplication: more passing tests (no UK/PK) #8225
    • Online DDL: reject ALTER TABLE...RENAME statements #8227
    • Online DDL/VReplication: fail on existence of FOREIGN KEY #8228
    • Online DDL/VReplication: test PK column case change #8336
    • Online DDL/VReplication: add PK DATETIME->TIMESTAMP test #8338
    • Bugfix: assign cexpr.references for column CONVERTed to utf8mb4 #8355

    VReplication

    • Towards a VReplication/OnlineDDL testing suite #8181
    • Online DDL/Vreplication: column type awareness #8239
    • remove duplicate ReadMigrations, fixing build #8258
    • VReplication (and by product, Online DDL): support GENERATED column as part of PRIMARY KEY #8335

    vttestserver

    • docker/vttestserver: Set max_connections in default-fast.cnf at container build time #7810

    Documentation

    Build/CI

    • Update version for latest snapshot #7801
    • v10 GA Release Notes #7964
    • Updates and Corrections to v9 Release Notes #8295

    Query Serving

    • Update link to 'Why FK not supported in Online DDL' blog post #8372

    Other

    • correct several wrong words #7822
    • MAINTAINERS.md: Update enisoc's email. #7909
    • Modification of the Markdown list format in the release notes template #7962
    • Update vreplicator docs #8014
    • Release notes: add Known Issues #8027

    Enhancement

    Observability

    • [trace] Add logging support to opentracing plugins #8289

    Build/CI

    • Makefile: add cross-build target for cross-compiling client binaries #7806
    • git: improve signoff detection #7852
    • Release notes generation #7932
    • vitess/lite docker build fails for mysql80 #7943
    • Fix a gofmt warning #7959
    • docker/vttestserver/run.sh: Add $CHARSET environment variable #7970
    • docker/lite/install_dependencies.sh: If the dependency install loop reaches its max number of retries, consider that a failure; exit the script nonzero so the build halts. #7976
    • Add commit count and authors to the release notes #7982
    • Automate version naming and release tagging #8034
    • Make sure to only allow codegen when the rest of the code compiles #8169
    • Add options to vttestserver to pass -foreign_key_mode, -enable_online_ddl, and -enable_direct_ddl through to vtcombo #8177

    Cluster management

    • Dynamic throttle metric threshold #7742
    • Bump Bootstrap version, per CVE-2018-14040 (orchestrator/vtorc) #7824
    • Check tablet alias before removing after error stream #7915
    • vttablet/tabletmanager - add additional test for tmstate.Open() #7993
    • Add vttablet_restore_done hook #8007
    • Added ValidateVSchema #8012
    • Add RemoteOperationTimeout to both legacy and grpc ChangeTabletType implementations. #8052
    • [vtctldserver] Add guard against self-reparent, plus misc updates #8084
    • [tm_state] updateLocked should re-populate local metadata tables to reflect promotion rule changes #8107
    • [vtctld] Migrate ApplyVSchema to VtctldServer #8113
    • vtctl.generateShardRanges -> key.GenerateShardRanges #8134
    • etcd: add grpc.WithBlock to client config #8205
    • [workflow] Add tracing to GetWorkflows endpoint #8266
    • [vtctldclient] Add legacy shim #8284
    • [vtctldserver] Add tracing #8285
    • [vtctldserver] Add additional backup info fields #8321
    • [workflow] Call scanWorkflow concurrently #8272

    Query Serving

    • [Gen4] Implemented table alias functionality in the semantic analysis #7629
    • Improved error messages (tabletserver) #7747
    • Allow modification of tablet unhealthy_threshold via debugEnv #7753
    • [Gen4] Initial Horizon Planning #7775
    • [tablet, queryrules] Extend query rules to check comments #7784
    • Add support for showing global gtid executed per shard #7856
    • Minor cleanups around errors on the vtgate #7864
    • log unsupported queries #7865
    • Show databases like #7912
    • Add rank as reserved keyword #7944
    • Online DDL: introducing ddl_strategy -singleton-context flag #7946
    • Ignore the error and log as warn if not able to validate the current system setting for check and ignore case #8004
    • Detect and signal schema changes on vttablets #8005
    • DDL bypass plan #8013
    • Online DDL: progress & ETA for Vreplication migrations #8015
    • Scatter errors as warning in olap query #8018
    • livequeryz: livequeryz/terminate link should be relative #8025
    • Handle online DDL user creation if we do not have SUPER privs (e.g. AWS Aurora) #8038
    • Update the schema copy with minimal changes #8067
    • Gen4: Support for order by column not present in projection #8070
    • Schema tracking in vtgate #8074
    • protobuf: upgrade #8075
    • added mysql 8.0 reserved keywords #8086
    • Fix ghost/pt-osc in the external DB case where MySQL might be reporting #8115
    • Support for vtgate -enable_direct_ddl flag #8116
    • add transaction ID to query log for acquisition and analysis #8133
    • Inline reference #8136
    • Improve ScatterErrorsAsWarnings functionality #8139
    • Gen4: planning Select IN #8155
    • Primary key name #8188
    • Online DDL: read and publish gh-ost FATAL message where possible #8192
    • Expose inner net.Conn to be able to write better unit tests #8217
    • vtgate: validate important flags during startup #8218
    • Online DDL/VReplication: AUTO_INCREMENT support and tests #8223
    • [Gen4] some renaming from v4 to Gen4 #8234
    • Schema Tracking: Optimize and Bug Fix #8243
    • gen4: minor refactoring to improve readability #8245
    • gen4: plan more opcodes #8254
    • Online DDL: report rows_copied for migration (initial support in gh-ost) #8255
    • Schema tracking: new tables in sharded keyspace #8256
    • [parser] use table_alias for ENGINE option in CREATE TABLE stmt #8307
    • Add /debug/env for vtgate #8292
    • gen4: outer joins #8312
    • Gen4: expand star in projection list #8325
    • gen4: Fail all queries not handled well by gen4 #8359
    • Gen4 fail more2 #8382

    VReplication

    • Use Dba user when Vexec is runAsAdmin #7731
    • VReplication: add table for logging stream errors, state changes and key workflow steps #7831
    • Fix some static check warning #7960
    • VSchema Validation on ReshardWorkflow Creation #7977
    • VReplication: tracking rows_copied #7980
    • Vdiff formatting improvements #8079
    • VReplication: ignore generated columns in workflows #8129
    • VReplication Copy Phase: Increase default replica lag tolerance. Also make it and copy timeout modifiable via flags #8130
    • Materialize: Add additional comparison operators in Materialize and fix bug where they not applied for sharded keyspaces #8247
    • Copy Phase: turn on OptimizeInserts by default #8248
    • Tracker/VStreamer: only reload schema for tables in current database and not for internal table artifacts #8257
    • Online DDL/Vreplication suite: support ENUM->VARCHAR/TEXT type change #8275
    • Added TableName to DiffReport struct. #8279
    • New VReplication lag metric #8306
    • VStream API: add heartbeat for idle streams #8244
    • Online DDL/VReplication: support non-UTF8 character sets #8322
    • Online DDL/Vreplication suite: fix test for no shared UK #8334
    • Online DDL/VReplication: support DROP+ADD column of same name #8337
    • Online DDL/VReplication test suite: support ENUM as part of PRIMARY KEY #8345

    VTAdmin

    • [vtadmin-web] Add useSyncedURLParam hook to persist filter parameter in the URL #7857
    • [vtadmin-web] Display more data on /gates view and add filtering #7876
    • [vtadmin] Promote ErrNoSchema to a TypedError which returns http 404 #7885
    • [vtadmin] gate/tablet/vtctld FQDNs #7886
    • [vtadmin-web] Display vindex data on Schema view #7917
    • [vtadmin-web] Add filtering and source/target shards to Workflows view #7948
    • [vtadmin-web] Add filtering + shard counts/status to Keyspaces view #7991
    • [vtadmin-web] Display shard state on Tablets view + extract tablet utilities #7999
    • [vtadmin] experimental tabletdebug #8003
    • [vtadmin-web] Display timestamps + stream counts on Workflows view #8009
    • [vtadmin-web] Updates to table styling + other CSS #8072
    • [vtadmin-web] Add Tab components #8119
    • [vtadmin-api] Update GetTablet to use alias instead of hostname #8163
    • [vtadmin-api] Rename flag 'http-tablet-fqdn-tmpl' to 'http-tablet-url-tmpl' + update vtadmin flags for local example #8164
    • Add --tracer flag to vtadmin and actually start tracing #8165
    • [vtadmin-web] The hastiest Tablet view (+ experimental debug vars) #8170
    • [vtadmin-web] Add tabs to Workflow view #8203
    • [vtctld] Add GetSrvVSchemas command #8221
    • [vtadmin-api] Add HTTP endpoints for /api/srvvschemas and /api/srvvschema/{cluster}/{cell} #8226
    • [vtadmin-web] Add QPS and VReplicationQPS charts to Tablet view #8263
    • [vtadmin-web] Add client-side error handling interface + Bugsnag implementation #8287
    • [vtadmin-web] Add chart for stream vreplication lag across all streams in a workflow #8331
    • [vtctldproxy] Add more annotations to vtctld Dial calls #8346

    vttestserver

    • Vttest create db #7989
    • Adds an environment variable to set the MySQL max connections limit in vttestserver docker image #8210

    Feature Request

    Build/CI

    • [trace] Add optional flags to support flexible jaeger configs #8199

    Cluster management

    • Add VtctldServer to vtcombo #7896
    • [vtctldserver] Migrate routing rules RPCs, and also RebuildVSchemaGraph #8197
    • [vtctldserver] Migrate CellInfo, CellAlias rw RPCs #8219
    • [vtctldserver] Add RefreshState RPCs #8232

    Query Serving

    • VTGate grpc implementation of Prepare and CloseSession #8211
    • Schema tracking: One schema load at a time per keyspace #8224

    VTAdmin

    • [vtadmin-web] Add DataFilter + Workspace layout components #8032
    • [vtadmin-web] Add Tooltip + HelpTooltip components #8076
    • [vtadmin-web] Add initial Stream view, render streams on Workflow view #8091
    • [vtadmin-web] The hastiest-ever VTExplain UI #8092
    • [vtadmin-web] Add source-map-explorer util #8093
    • [vtadmin-web] Add Keyspace detail view #8111
    • [vtadmin-api] Add GetKeyspace endpoint #8125
    • [workflow] Add vreplication_log data to workflow protos, and VtctldServer.GetWorkflows method #8261
    • [vtadmin] Add debug endpoints #8268

    Internal Cleanup

    Build/CI

    • Makefile: fix cross-build comments #8246
    • remove unused hooks with refs to master branch #8250
    • [vtctldserver] Update tests to prevent races during tmclient init #8320
    • sks-keyservers.net seems to be finally dead, replace with #8363
    • endtoend: change log level of etcd start from error to info #8370

    Cluster management

    • Online DDL: code cleanup #7589
    • [wrangler|topotools] Migrate UpdateShardRecords, RefreshTabletsByShard, and {Get,Save}RoutingRules #7965
    • Tear down old stream_migrater shim, now that we're fully in package workflow #8073
    • [mysqlctl] Restructure MetadataManager to reduce public API surface area #8152
    • naming: master to primary #8251
    • vtorc: code cleanup #8269

    Query Serving

    • Plan StreamExecute Queries #7941

    VReplication

    • [workflow] extract migration targets from wrangler #7934
    • [wrangler|workflow] Extract vrStream type to workflow.VReplicationStream #7966
    • [wrangler|workflow] Extract workflowState and workflowType out to package workflow #7967
    • [wrangler|workflow] extract *wrangler.streamMigrater to workflow.StreamMigrator #8008
    • [workflow] Migrate getCellsWith{Shard,Table}ReadsSwitched, TrafficSwitchDirection and TableRemovalType to package workflow #8190
    • [workflow] Cleanup wrangler wrappers, migrate checkIfJournalExistsOnTablet to package workflow #8193

    VTAdmin

    • [vtadmin-api] Replace magic numbers with net/http constants #8127
    • [vtadmin-web] Move single-entity view components into subfolders #8202
    • [vtadmin] Ensure we log any errors when closing the tracer #8262

    Other

    Build/CI

    • add vtctldclient to binary directory #7889

    Cluster management

    • vttablet/tabletmanager: add isInSRVKeyspace/isShardServing #7929

    Other

    • Change VitessInputFormat key type #199
    • Online DDL plan via Send; "singleton" migrations on tablets #7785
    • flaky onlineddl tests: reduce -online_ddl_check_interval #7847
    • Looking into flaky endtoend upgrade test #7900
    • fixing flaky upgrade test #7901

    Query Serving

    • Introduce Concatenated Fixed-width Composite or CFC vindex #7537
    • Add common tags for stats backends that support it #7651
    • Add support for showing global vgtid executed #7797
    • perf: vttablet/mysql optimizations #7800
    • Fix bug with reserved connections to stale tablets #7879
    • Memory Sort to close the goroutines when callback returns error #7903
    • OnlineDDL: more migration check ticks upon migration start #7961
    • Update gh-ost binary to v1.1.3 #8021
    • VReplication Online DDL: fix classification of virtual columns #8043

    VReplication

    • VReplicationErrors metric: use . as delimiter instead of _ to behave well with Prometheus #7807

    VTAdmin

    • Update GetSchema filtering to exclude shards where IsMasterServing but no MasterAlias #7805
    • [vtadmin-api] Reintroduce include_non_serving_shards opt to GetSchema #7814
    • [vtadmin-web] Add DataCell component #7817
    • Rewrite useTableDefinitions hook as getTableDefinitions util #7821
    • [vtadmin-web] Display (approximate) table sizes + row count on /schemas view #7826
    • [vtadmin-web] Add Pip + TabletServingPip components #7827

    Performance

    Query Serving

    • vttablet: stream consolidation #7752
    • perf: optimize bind var generation #7828

    VReplication

    • VReplication: optimize the catchup phases by filtering out rows which not within range of copied pks during inserts #7708
    • VReplication: ability to compress gtid when stored in _vt.vreplication's pos column #7877
    • vreplication: performance & benchmarks (table copying) #7881
    • vreplication: dynamic packet sizing #7933
    • perf: vreplication client CPU usage #7951
    • VDIff: fix performance regression introduced by progress logging #8016
    • proto: Generate faster code using vtprotobuf #8173
    • proto: enable pooling for vreplication #8273

    Testing

    Build/CI

    • Attempt to fix TLS Server Flaky test #7842
    • TestMakeCommonTags Flaky Test: match elements #7843
    • upgrade tests: test against v9.0.0 #7848
    • FOSSA scan added #7862
    • ci: fix all racy tests #7904
    • CI: Revert docker change for unit tests from #7868 #7940
    • Add online ddl on start queries to schema list in vtexplain tablet #8397

    Java

    • [Java] JDBC mysql driver test #8154

    Other

    • Fuzzing: Fixup oss-fuzz build script #7782
    • Wrangler tests: Return a fake tablet in the wrangler test dialer to avoid tablet picker errors spamming the test logs #7863

    Query Serving

    • clean up test #7816
    • tabletserver: fix flaky test #7851
    • Planbuilder: Add fuzzer #7902
    • mysql: Small adjustments to fuzzer #7907
    • vtgate/engine: Add fuzzer #7914
    • Adding Fuzzer Test Cases #8106
    • Addition of fuzzer issues #8195

    VReplication

    • vstreamer: Add fuzzer #7918

    vttestserver

    • Speedup new vttestserver tests #8229
    • Vttestserver docker test #8253

    Cluster management

    • Make timestamp authoritative for master information #8381

    The release includes 1041 commits (excluding merges)

    Thanks to all our contributors: @AdamKorcz, @GuptaManan100, @Hellcatlk, @Johnny-Three, @acharisshopify, @ajm188, @alexrs, @aquarapid, @askdba, @deepthi, @dependabot[bot], @doeg, @dyv, @enisoc, @frouioui, @gedgar, @guidoiaquinti, @harshit-gangal, @hkdsun, @idvoretskyi, @kirs, @mcronce, @narcsfz, @noxiouz, @rafael, @rohit-nayak-ps, @setassociative, @shlomi-noach, @systay, @tokikanno, @vmg, @wangmeng99, @yangxuanjia, @zhangshj-inspur

    Source code(tar.gz)
    Source code(zip)
    vitess-11.0.0-rc1-740c379.tar.gz(396.98 MB)
    vitess-11.0.0_rc1-740c379.x86_64.rpm(394.71 MB)
    vitess_11.0.0-rc1-740c379_amd64.deb(396.99 MB)
  • v10.0.2(May 27, 2021)

    Bug fixes

    Query Serving

    • Fixes encoding of sql strings #8029
    • Fix for issue with information_schema queries with both table name and schema name predicates #8099
    • PRIMARY in index hint list for release 10.0 #8159

    VReplication

    • VReplication: Pad binlog values for binary() columns to match the value returned by mysql selects #8137

    CI/Build

    Build/CI

    • update release notes with known issue #8081

    Documentation

    Other

    • Post v10.0.1 updates #8045

    Enhancement

    Build/CI

    • Added release script to the makefile #8030

    Other

    • Add optional TLS feature to gRPC servers #8176

    Other

    Build/CI

    • Release 10.0.1 #8031

    The release includes 14 commits (excluding merges) Thanks to all our contributors: @GuptaManan100, @askdba, @deepthi, @harshit-gangal, @noxiouz, @rohit-nayak-ps, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.2-fc78470.tar.gz(386.97 MB)
    vitess-10.0.2-fc78470.x86_64.rpm(384.69 MB)
    vitess_10.0.2-fc78470_amd64.deb(386.96 MB)
  • v9.0.2(May 25, 2021)

    Bug fixes

    Cluster management

    • Restore: Check disable_active_reparents properly before waiting for position update #8114

    Query Serving

    • Fix information_schema query with system schema in table_schema filter #8095
    • Fix for issue with information_schema queries with both table name and schema name predicates #8096
    • Fix for transactions not allowed to finish during PlannedReparentShard #8098
    • PRIMARY in index hint list #8158

    CI/Build

    Build/CI

    • Release 9.0.1 #8065
    • 9.0.0: update release notes with known issue #8080 #8082
    • Added release script to the makefile #8182
    • Update do_release to work in the 9.0 branch #8184

    Performance

    Cluster management

    • Revert "backup: Use pargzip instead of pgzip for compression." #8174

    The release includes 17 commits (excluding merges) Thanks to all our contributors: @GuptaManan100, @deepthi, @Harshit, @rafael, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-9.0.2-58cd7b0.tar.gz(318.84 MB)
    vitess-9.0.2-58cd7b0.x86_64.rpm(316.98 MB)
    vitess_9.0.2-58cd7b0_amd64.deb(318.85 MB)
  • v9.0.1(May 6, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Bug fixes

    Cluster management

    • Backport: Respect -disable_active_reparents in backup/restore #8063

    Other

    • [9.0] fix regression - should be able to plan subquery on top of subquery #7683

    Query Serving

    • [9.0] Fix for reserved connection usage with transaction #7666
    • [9.0] Fix MySQL Workbench failure on login with select current_user() #7706
    • [9.0] make sure to handle subqueries on top of subqueries #7776
    • [9.0] make sure to not log sensitive information #7778
    • [9.0] ddl bypass planner #8035
    • [9.0] Memory Sort to close the goroutines when callback returns error #8040
    • [9.0] Fix bug with reserved connections to stale tablets #8041
    • [9.0] Fix for Query Serving when Toposerver is Down #8046
    • ignore the error and log as warn if not able to validate the current system settings value #8062

    VReplication

    • VReplication: fix vreplication timing metrics #8060
    • VReplication: Pad binlog values for binary() columns to match the value returned by mysql selects #8061

    Documentation

    Other

    • 9.0.0 Release Notes #7384

    Other

    Build/CI

    • Fix Dockerfiles for vtexplain and vtctlclient #7418

    Query Serving

    • Fix table parsing on VSchema generation #7511
    • [9.0] Show anywhere plan fix to consider default keyspace #7530
    • [9.0] Reset Session for Reserved Connection when the connection id is not found #7544
    • Healthcheck: update healthy tablets correctly when a stream returns an error or times out #7732

    VReplication

    • MoveTables: Refresh SrvVSchema and source tablets on completion #8059

    The release includes 43 commits (excluding merges) Thanks to all our contributors: @askdba, @deepthi, @dyv, @harshit-gangal, @rafael, @rohit-nayak-ps, @shlomi-noach, @systay

    Source code(tar.gz)
    Source code(zip)
    vitess-9.0.1-42c38e5.tar.gz(318.28 MB)
    vitess-9.0.1-42c38e5.x86_64.rpm(316.42 MB)
    vitess_9.0.1-42c38e5_amd64.deb(318.28 MB)
  • v10.0.1(May 4, 2021)

  • v10.0.0(Apr 27, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Known Issues

    • Running binaries with --version or running select @@version from a MySQL client still shows 10.0.0-RC1
    • Online DDL cannot be used if you are using the keyspace filtering feature of VTGate
    • VReplication errors when a fixed-length binary column is used as the sharding key #8080

    Bugs Fixed

    VTGate / MySQL compatibility

    • Remove printing of ENFORCED word so that statements remain compatible with mysql 5.7 #7458
    • Allow any ordering of generic options in column definitions #7459
    • Corrects the comment handling in vitess #7581
    • Fix regression - should be able to plan subquery on top of subquery #7682
    • Nullable Timestamp Column Fix #7740
    • VTGate: Fix the error messages in drop, create and alter database commands #7397
    • VTGate: Fix information_schema query with system schema in table_schema filter #7430
    • VTGate: Fix Set Statement in Tablet when executed with bindvars #7431
    • VTGate: Fix for Query Serving when Toposerver is Down #7484
    • VTGate: Add necessary bindvars when preparing queries #7493
    • VTGate: Show anywhere plan fix to consider default keyspace #7531
    • VTGate: Fix table parsing on VSchema generation #7511
    • VTGate: Use the emulated MySQL version for special comments #7510
    • VTGate: Reset Session for Reserved Connection when the connection id is not found #7539
    • VTGate: Healthcheck: update healthy tablets correctly when a stream returns an error or timeout #7732
    • VTGate: Fix for reserved connection usage with transaction #7646
    • VTGate: Fix MySQL Workbench failure on login with select current_user() #7705
    • VTGate: Constraint names and database names with spaces. #7745
    • VTGate: Fix dual table query when system schema is selected database #7734

    Other

    • VTTablet: Correctly initialize statsTabletTypeCounts during startup #7390
    • Backup/Restore: Respect -disable_active_reparents in backup/restore #7576
    • Backup/Restore: check disable_active_reparents properly before waiting for position update #7703

    Functionality Added or Changed

    VTGate / MySQL compatibility / Query Serving

    • VTGate: Gen4 Planner: AxB vs BxA #7274
    • VTGAte: Gen4 fallback planning #7370
    • VTGate: Support for CALL procedures #7287
    • VTGate: Set default for @@workload to OLTP #7288
    • VTGate: Change @@version and @@version_comment #7337
    • VTGate: Fix VitessAware system variables of type boolean return NULL when MySQL is not involved #7353
    • VTGate: Add stats for RowsAffected similar to RowsReturned #7380
    • VTGate: Added information_schema_stats_expiry to allowed list of set vars #7435
    • VTGate: LFU Cache Implementation #7439
    • VTGate: Describe table to route based on table name and qualifier #7445
    • VTGate: Olap error message fix #7448
    • VTGate: Temporary Table support in unsharded keyspace #7411
    • VTGate: Publish table size on schema #7444
    • VTGate: Support for caching_sha2_password plugin in mysql/client #6716
    • VTGate: Moving Show plan from executor to planbuilder #7475
    • VTGate: Adds another case to merge routes for information_schema queries #7504
    • VTGate: Add innodb_read_rows as vttablet metric #7520
    • VTGate: Adds support for Show variables #7547
    • VTGate: gen4: fail unsupported queries #7409
    • VTGate: Fix Metadata in SHOW queries #7540
    • VTGate: Update AST helper generation #7558
    • VTGate: Avoiding addition of redundant unary operators #7579
    • VTGate: Optimise AST rewriting #7583
    • VTGate: Add Show Status query to vtexplain and make asthelpergen/sizegen quiet #7590
    • VTGate: Add support for SELECT ALL #7593
    • VTGate: Empty statement error code change in sql parsing #7618
    • VTGate: Socket system variable to return vitess mysql socket #7637
    • VTGate: Make DROP/CREATE DATABASE pluggable #7381
    • VTGate: Allow Select with lock to pass through in vttablet #7584
    • VTGate: Fix ordering in SELECT INTO and printing of strings #7655
    • VTGate: AST Equals code generator #7672
    • VTGate: [tabletserver] More resilient wait for schema changes #7684
    • VTGate: Fix flush statement planner #7695
    • VTGate: Produce query warnings for using features not supported when sharded #7538
    • VTGate: Support for ALTER VITESS_MIGRATION statements #7663
    • VTGate: Solve I_S queries using CNF rewriting #7677
    • VTGate: System schema queries #7685
    • VTGate: Make the AST visitor faster #7701
    • VTGate: COM_PREPARE - Single TCP response packet with all MySQL Packets #7713
    • VTGate: Replace the database name in result fields only if needed #7714
    • VTGate: Split ast_helper into individual gen files #7727
    • VTGate: Adds support for ordering on character fields for sharded keyspace queries #7678
    • VTGate: Show columns query on system schema #7729
    • VTGate: Disallow foreign key constraint on ddl #7780
    • VTGate: VTGate: support -enable_online_ddl flag #7694
    • VTGate: Default to false for system settings to be changed per session at the database connection level #7921
    • VTGate: vtctl: return error on invalid ddl_strategy #7924
    • VTGate: [10.0] Squashed backport of #7903 #7927
    • VTGate: [10.0] Fix bug with reserved connections to stale tablets #7935
    • VTGate: [10.0] Fix for keyspaces_to_watch regression #7936
    • VTGate: [10.0] Update healthy tablets correctly for primary down #7937
    • VTGate: [10.0] Allow modification of tablet unhealthy_threshold via debugEnv #7938

    Testing

    • Fuzzing: Add vtctl fuzzer #7605
    • Fuzzing: Add more fuzzers #7622
    • Fuzzing: Add 3 fuzzers for mysql endpoints #7639
    • Fuzzing: Add oss-fuzz build script #7591
    • Fuzzing: Add requirement for minimum length of input data #7722
    • Fuzzing: Add new mysql fuzzer #7660
    • Fuzzing: Add [grpcvtgateconn] fuzzer #7689
    • Fuzzing: Make mysql fuzzer more calls during each iteration #7766

    Performance

    • VTGate: [perf] zero-copy tokenizer #7619
    • VTGate: [perf: sqlparser faster formatting #7710
    • VTGate :[perf] Cache reserved bind variables in queries #7698
    • VTGate: [perf] sqlparser yacc codegen #7669
    • VTGate: Making fast AST rewriter faster #7726
    • VTGate: Cached Size Implementation #7387
    • VTGate: Plan remove mutexes #7468
    • LFU Cache Bug Fixes #7479
    • [cache] Handle all possible initialization cases #7556
    • VTGate: [servenv] provide a global flag for profiling #7496
    • VTGate: [vttablet] Benchmarks and two small optimizations #7560
    • [pprof]: allow stopping profiling early with a signal #7594
    • perf: RPC Serialization #7519
    • perf: keyword lookups in the tokenizer #7606

    Cluster Management

    • [vtctld] Migrate topo management RPCs #7395
    • [vtctldclient] Set SilenceErrors on the root command, so we don't double-log #7404
    • [vtctldclient] Command line flags: dashes and underscores synonyms #7406
    • Extract the maxReplPosSearch struct out to topotools #7420
    • Add protoutil package, refactor ISP to use it #7421
    • Add ErrorGroup to package concurrency, use in waitOnNMinusOneTablets #7429
    • [vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer #7434
    • [vtctld/wrangler] Extract EmergencyReparentShard logic to dedicated struct and add unit tests #7464
    • Provide named function for squashing usage errors; start using it #7451
    • [concurrency] Add guard against potentially blocking forever in ErrorGroup.Wait() when NumGoroutines is 0 #7463
    • Add hook for statsd integration #7417
    • [concurrency] Add guard against potentially blocking forever in ErrorGroup.Wait() when NumGoroutines is 0 #7463
    • Resilient rebuild keyspace graph check, tablet controls not in RebuildKeyspaceGraph command #7442
    • [reparentutil / ERS] confirm at least one replica succeeded to SetMaster, or fail #7486
    • [reparentutil / wrangler] Extract PlannedReparentShard logic from wrangler to PlannedReparenter struct #7501
    • Add backup/restore duration stats #7512
    • Refresh replicas and rdonly after MigrateServedTypes except on skipRefreshState. #7327
    • [eparentutil] ERS should not attempt to WaitForRelayLogsToApply on primary tablets that were not running replication #7523
    • [orchestrator] prevent XSS attack via 'orchestrator-msg' params #7526
    • [vtctld] Add remaining reparent commands to VtctldServer #7536
    • [reparentutil] ERS should not attempt to WaitForRelayLogsToApply on primary tablets that were not running replication #7523
    • Shutdown vttablet gracefully if tablet record disappears #7563
    • ApplySchema: -skip_preflight #7587
    • Table GC: disable binary logging on best effort basis #7588
    • Addition of waitSig pprof argument to start recording on USR1 #7616
    • Add combine TLS certs feature #7609
    • Check error response before attempting to access InitShardPrimary response #7658
    • [vtctld] Migrate GetSrvKeyspace as GetSrvKeyspaces in VtctldServer #7680
    • [vtctld] Migrate ShardReplicationPositions #7690
    • [reparentutil | ERS] Bind status variable to each goroutine in WaitForRelayLogsToApply #7692
    • [servenv] Fix var shadowing caused by short variable declaration #7702
    • [vtctl|vtctldserver] List/Get Tablets timeouts #7715
    • vtctl ApplySchema supports '-request_context' flag #7777

    VReplication

    • VReplication: vstreamer to throttle on source endpoint #7324
    • VReplication: Throttle on target tablet #7364
    • VReplication: Throttler: fix to client usage in vreplication and table GC #7422
    • VReplication: MoveTables/Reshard add flags to start workflows in a stopped state and to stop workflow once copy is completed #7449
    • VReplication: Support for caching_sha2_password plugin in mysql/client #6716
    • VReplication: Validate SrvKeyspace during Reshard/SwitchReads #7481
    • VReplication: [MoveTables] Refresh SrvVSchema (for Routing Rules) and source tablets (for Blacklisted Tables) on completion #7505
    • VReplication : Data migration from another Vitess cluster #7546
    • VReplication : [materialize] Add cells and tablet_types parameters #7562
    • VReplication: JSON Columns: fix bug where vreplication of update statements were failing #7640
    • VReplication: Make the frequency at which heartbeats update the _vt.vreplication table configurable #7659
    • VReplication: Error out if binlog compression is turned on #7670
    • VReplication: Tablet throttler: support for custom query & threshold #7541
    • VStream API: allow aligning streams from different shards to minimize skews across the streams #7626
    • VReplication: Backport 7809: Update rowlog for the API change made for the vstream skew alignment feature #7890

    OnlineDDL

    • OnlineDDL: update gh-ost binary to v1.1.1 #7394
    • Online DDL via VReplication #7419
    • Online DDL: VReplicatoin based mini stress test CI #7492
    • OnlineDDL: Revert for VReplication based migrations #7478
    • Online DDL: Internal support for eta_seconds #7630
    • Online DDL: Support 'SHOW VITESS_MIGRATIONS' queries #7638
    • Online DDL: Support for REVERT VITESS_MIGRATION statement #7656
    • Online DDL: Declarative Online DDL #7725

    VTAdmin

    Vitess 10.0 introduces a highly-experimental multi-cluster admin API and web UI, called VTAdmin. Deploying the vtadmin-api and vtadmin-web components is completely opt-in. If you're interested in trying it out and providing early feedback, come find us in #feat-vtadmin in the Vitess slack. Note that VTAdmin relies on the new VtctldServer API, so you must be running the new grpc-vtctld service on your vtctlds in order to use it.

    • VTAdmin: Add vtadmin-web build flag for configuring fetch credentials #7414
    • VTAdmin: Add cluster field to vtadmin-api's /api/gates response #7425
    • VTAdmin: Add /api/clusters endpoint to vtadmin-api #7426
    • VTAdmin: Add /api/schemas endpoint to vtadmin-api #7432
    • VTAdmin: [vtadmin-web] Add routes and simple tables for all entities #7440
    • VTAdmin: [vtadmin-web] Set document.title from route components #7450
    • VTAdmin: [vtadmin-web] Add DataTable component with URL pagination #7487
    • VTAdmin: [vtadmin-api] Add shards to /api/keyspaces response #7453
    • VTAdmin: [vtadmin-web] Add replaceQuery + pushQuery to useURLQuery hook #7507
    • VTAdmin: [vtadmin-web] An initial pass for tablet filters #7515
    • VTAdmin: [vtadmin-web] Add a Select component #7524
    • VTAdmin: [vtadmin-api] Add /vtexplain endpoint #7528
    • VTAdmin: [vtadmin-api] Reorganize tablet-related functions into vtadmin/cluster/cluster.go #7553
    • VTAdmin: Three small bugfixes in Tablets table around stable sort order, display type lookup, and filtering by type #7568
    • VTAdmin: [vtadmin] Add GetSchema endpoint #7596
    • VTAdmin: [vtadmin/testutil] Add testutil helper to manage the complexity of recursively calling WithTestServer #7601
    • VTAdmin: [vtadmin] Add FindSchema route #7610
    • VTAdmin: [vtadmin-web] Add simple /schema view with table definition #7615
    • VTAdmin: [vtadmin] vschemas api endpoints #7625
    • VTAdmin: [vtadmin] Add support for additional service healthchecks in grpcserver #7635
    • VTAdmin: [vtadmin] test refactors #7641
    • VTAdmin: [vtadmin] propagate error contexts #7642
    • VTAdmin: [vtadmin] tracing refactor #7649
    • VTAdmin: [vtadmin] GetWorkflow(s) endpoints #7662
    • VTAdmin: [vitessdriver|vtadmin] Support Ping in vitessdriver, use in vtadmin to healthcheck connections during Dial #7709
    • VTAdmin: [vtadmin] Add to local example #7699
    • VTAdmin: [vtexplain] lock #7724
    • VTAdmin: [vtadmin] Aggregate schema sizes #7751
    • VTAdmin: [vtadmin-web] Add comments + 'options' parameter to API hooks #7754
    • VTAdmin: [vtadmin-web] Add common max-width to infrastructure table views #7760
    • VTAdmin: [vtadmin-web] Add hooks + skeleton view for workflows #7762
    • VTAdmin: [vtadmin-web] Add a hasty filter input to the /schemas view #7779

    Other / Tools

    • [rulesctl] Implements CLI tool for rule management #7712

    Examples / Tutorials

    • Source correct shell script in README #7749

    Documentation

    • Add Severity Labels document #7542
    • Remove Google Groups references #7664
    • Move some commas around in README.md :) #7671
    • Add Andrew Mason to Maintainers List #7757

    Build/CI Environment Changes

    • Update java build versions to vitess 10.0.0 #7383
    • CI: check run analysis to post JSON from file #7386
    • Fix Dockerfiles for vtexplain and vtctlclient #7418
    • CI: Add descriptive names to vrep shards. Update test generator script #7454
    • CI: adding 'go mod tidy' test #7461
    • Docker builds vitess/vtctlclient to install curl #7466
    • Add VT_BASE_VER to vtexplain/Dockerfile #7467
    • Enable -mysql_server_version in vttestserver, and utilize it in vttestserver container images #7474
    • [vtctld | tests only] testtmclient refactor #7518
    • CI: skip some tests on forked repos #7527
    • Workflow to check make sizegen #7535
    • Add mysqlctl docker image #7557
    • Restore CI workflow shard 26, accidentally dropped #7569
    • Update CODEOWNERS #7586
    • CI: ci-workflow-gen turn string to array to reduce conflicts #7582
    • Add percona-toolkit (for pt-osc/pt-online-schema-change) to the docker/lite images #7603
    • CI: Use ubuntu-18.04 in tests #7614
    • [vttestserver] Fix to work with sharded keyspaces #7617
    • Add tools.go #7517
    • Make vttestserver compatible with persistent data directories #7718
    • Add vtorc binary for rpm,deb builds #7750
    • Fixes bug that prevents creation of logs directory #7761
    • [Java] Guava update to 31.1.1 #7764
    • make: build vitess as static binaries by default #7795 ← Potentially breaking change
    • make: build vitess as static binaries by default (10.0 backport) #7808
    • java: prepare java version for release 10.0 #7922

    Functionality Neutral Changes

    • VTGate: Remove unused key.Destination.IsUnique() #7565
    • VTGate: Add information_schema query on prepare statement #7746
    • VTGate: Tests for numeric_precision and numeric_scale columns in information_schema #7763
    • Disable flaky test until it can be fixed #7623
    • Tests: reset stat at the beginning of test #7644
    • Cleanup mysql server_test #7645
    • vttablet: fix flaky tests #7543
    • Removed unused tests for Wordpress installation #7516
    • Fix unit test fail after merge #7550
    • Add test with NULL input values for vindexes that did not have any. #7552

    VtctldServer

    As part of an ongoing effort to transition from the VtctlServer gRPC API to the newer VtctldServer gRPC API, we have updated the local example to use the corresponding new vtctldclient to perform InitShardPrimary (formerly, InitShardMaster) operations.

    To enable the new VtctldServer in your vtctld components, update the -service_map flag to include grpc-vtctld. You may specify both grpc-vtctl,grpc-vtctld to gracefully transition.

    The migration is still underway, but you may begin to transition to the new client for migrated commands. For a full listing, refer either to proto/vtctlservice.proto or consult vtctldclient --help.

    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.0-48dccf5.tar.gz(386.88 MB)
    vitess-10.0.0-48dccf5.x86_64.rpm(384.60 MB)
    vitess_10.0.0-48dccf5_amd64.deb(386.89 MB)
  • v10.0.0-rc1(Apr 6, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    The following PRs made changes to behaviors that clients might rely on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.

    Bugs Fixed

    VTGate / MySQL compatibility

    • Remove printing of ENFORCED word so that statements remain compatible with mysql 5.7 #7458
    • Allow any ordering of generic options in column definitions #7459
    • Corrects the comment handling in vitess #7581
    • Fix regression - should be able to plan subquery on top of subquery #7682
    • Nullable Timestamp Column Fix #7740
    • VTGate: Fix the error messages in drop, create and alter database commands #7397
    • VTGate: Fix information_schema query with system schema in table_schema filter #7430
    • VTGate: Fix Set Statement in Tablet when executed with bindvars #7431
    • VTGate: Fix for Query Serving when Toposerver is Down #7484
    • VTGate: Add necessary bindvars when preparing queries #7493
    • VTGate: Show anywhere plan fix to consider default keyspace #7531
    • VTGate: Fix table parsing on VSchema generation #7511
    • VTGate: Use the emulated MySQL version for special comments #7510
    • VTGate: Reset Session for Reserved Connection when the connection id is not found #7539
    • VTGate: Healthcheck: update healthy tablets correctly when a stream returns an error or timeout #7732
    • VTGate: Fix for reserved connection usage with transaction #7646
    • VTGate: Fix MySQL Workbench failure on login with select current_user() #7705
    • VTGate: Constraint names and database names with spaces. #7745
    • VTGate: Fix dual table query when system schema is selected database #7734

    Other

    • VTTablet: Correctly initialize statsTabletTypeCounts during startup #7390
    • Backup/Restore: Respect -disable_active_reparents in backup/restore #7576
    • Backup/Restore: check disable_active_reparents properly before waiting for position update #7703

    Functionality Added or Changed

    VTGate / MySQL compatibility / Query Serving

    • VTGate: Gen4 Planner: AxB vs BxA #7274
    • VTGAte: Gen4 fallback planning #7370
    • VTGate: Support for CALL procedures #7287
    • VTGate: Set default for @@workload to OLTP #7288
    • VTGate: Change @@version and @@version_comment #7337
    • VTGate: Fix VitessAware system variables of type boolean return NULL when MySQL is not involved #7353
    • VTGate: Add stats for RowsAffected similar to RowsReturned #7380
    • VTGate: Added information_schema_stats_expiry to allowed list of set vars #7435
    • VTGate: LFU Cache Implementation #7439
    • VTGate: Describe table to route based on table name and qualifier #7445
    • VTGate: Olap error message fix #7448
    • VTGate: Temporary Table support in unsharded keyspace #7411
    • VTGate: Publish table size on schema #7444
    • VTGate: Support for caching_sha2_password plugin in mysql/client #6716
    • VTGate: Moving Show plan from executor to planbuilder #7475
    • VTGate: Adds another case to merge routes for information_schema queries #7504
    • VTGate: Add innodb_read_rows as vttablet metric #7520
    • VTGate: Adds support for Show variables #7547
    • VTGate: gen4: fail unsupported queries #7409
    • VTGate: Fix Metadata in SHOW queries #7540
    • VTGate: Update AST helper generation #7558
    • VTGate: Avoiding addition of redundant unary operators #7579
    • VTGate: Optimise AST rewriting #7583
    • VTGate: Add Show Status query to vtexplain and make asthelpergen/sizegen quiet #7590
    • VTGate: Add support for SELECT ALL #7593
    • VTGate: Empty statement error code change in sql parsing #7618
    • VTGate: Socket system variable to return vitess mysql socket #7637
    • VTGate: Make DROP/CREATE DATABASE pluggable #7381
    • VTGate: Allow Select with lock to pass through in vttablet #7584
    • VTGate: Fix ordering in SELECT INTO and printing of strings #7655
    • VTGate: AST Equals code generator #7672
    • VTGate: [tabletserver] More resilient wait for schema changes #7684
    • VTGate: Fix flush statement planner #7695
    • VTGate: Produce query warnings for using features not supported when sharded #7538
    • VTGate: Support for ALTER VITESS_MIGRATION statements #7663
    • VTGate: Solve I_S queries using CNF rewriting #7677
    • VTGate: System schema queries #7685
    • VTGate: Make the AST visitor faster #7701
    • VTGate: COM_PREPARE - Single TCP response packet with all MySQL Packets #7713
    • VTGate: Replace the database name in result fields only if needed #7714
    • VTGate: Split ast_helper into individual gen files #7727
    • VTGate: Adds support for ordering on character fields for sharded keyspace queries #7678
    • VTGate: Show columns query on system schema #7729
    • VTGate: Disallow foreign key constraint on ddl #7780
    • VTGate: VTGate: support -enable_online_ddl flag #7694

    Testing

    • Fuzzing: Add vtctl fuzzer #7605
    • Fuzzing: Add more fuzzers #7622
    • Fuzzing: Add 3 fuzzers for mysql endpoints #7639
    • Fuzzing: Add oss-fuzz build script #7591
    • Fuzzing: Add requirement for minimum length of input data #7722
    • Fuzzing: Add new mysql fuzzer #7660
    • Fuzzing: Add [grpcvtgateconn] fuzzer #7689
    • Fuzzing: Make mysql fuzzer more calls during each iteration #7766

    Performance

    • VTGate: [perf] zero-copy tokenizer #7619
    • VTGate: [perf: sqlparser faster formatting #7710
    • VTGate :[perf] Cache reserved bind variables in queries #7698
    • VTGate: [perf] sqlparser yacc codegen #7669
    • VTGate: Making fast AST rewriter faster #7726
    • VTGate: Cached Size Implementation #7387
    • VTGate: Plan remove mutexes #7468
    • LFU Cache Bug Fixes #7479
    • [cache] Handle all possible initialization cases #7556
    • VTGate: [servenv] provide a global flag for profiling #7496
    • VTGate: [vttablet] Benchmarks and two small optimizations #7560
    • [pprof]: allow stopping profiling early with a signal #7594
    • perf: RPC Serialization #7519
    • perf: keyword lookups in the tokenizer #7606

    Cluster Management

    • [vtctld] Migrate topo management RPCs #7395
    • [vtctldclient] Set SilenceErrors on the root command, so we don't double-log #7404
    • [vtctldclient] Command line flags: dashes and underscores synonyms #7406
    • Extract the maxReplPosSearch struct out to topotools #7420
    • Add protoutil package, refactor ISP to use it #7421
    • Add ErrorGroup to package concurrency, use in waitOnNMinusOneTablets #7429
    • [vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer #7434
    • [vtctld/wrangler] Extract EmergencyReparentShard logic to dedicated struct and add unit tests #7464
    • Provide named function for squashing usage errors; start using it #7451
    • [concurrency] Add guard against potentially blocking forever in ErrorGroup.Wait() when NumGoroutines is 0 #7463
    • Add hook for statsd integration #7417
    • [concurrency] Add guard against potentially blocking forever in ErrorGroup.Wait() when NumGoroutines is 0 #7463
    • Resilient rebuild keyspace graph check, tablet controls not in RebuildKeyspaceGraph command #7442
    • [reparentutil / ERS] confirm at least one replica succeeded to SetMaster, or fail #7486
    • [reparentutil / wrangler] Extract PlannedReparentShard logic from wrangler to PlannedReparenter struct #7501
    • Add backup/restore duration stats #7512
    • Refresh replicas and rdonly after MigrateServedTypes except on skipRefreshState. #7327
    • [eparentutil] ERS should not attempt to WaitForRelayLogsToApply on primary tablets that were not running replication #7523
    • [orchestrator] prevent XSS attack via 'orchestrator-msg' params #7526
    • [vtctld] Add remaining reparent commands to VtctldServer #7536
    • [reparentutil] ERS should not attempt to WaitForRelayLogsToApply on primary tablets that were not running replication #7523
    • Shutdown vttablet gracefully if tablet record disappears #7563
    • ApplySchema: -skip_preflight #7587
    • Table GC: disable binary logging on best effort basis #7588
    • Addition of waitSig pprof argument to start recording on USR1 #7616
    • Add combine TLS certs feature #7609
    • Check error response before attempting to access InitShardPrimary response #7658
    • [vtctld] Migrate GetSrvKeyspace as GetSrvKeyspaces in VtctldServer #7680
    • [vtctld] Migrate ShardReplicationPositions #7690
    • [reparentutil | ERS] Bind status variable to each goroutine in WaitForRelayLogsToApply #7692
    • [servenv] Fix var shadowing caused by short variable declaration #7702
    • [vtctl|vtctldserver] List/Get Tablets timeouts #7715
    • vtctl ApplySchema supports '-request_context' flag #7777

    VReplication

    • VReplication: vstreamer to throttle on source endpoint #7324
    • VReplication: Throttle on target tablet #7364
    • VReplication: Throttler: fix to client usage in vreplication and table GC #7422
    • VReplication: MoveTables/Reshard add flags to start workflows in a stopped state and to stop workflow once copy is completed #7449
    • VReplication: Support for caching_sha2_password plugin in mysql/client #6716
    • VReplication: Validate SrvKeyspace during Reshard/SwitchReads #7481
    • VReplication: [MoveTables] Refresh SrvVSchema (for Routing Rules) and source tablets (for Blacklisted Tables) on completion #7505
    • VReplication : Data migration from another Vitess cluster #7546
    • VReplication : [materialize] Add cells and tablet_types parameters #7562
    • VReplication: JSON Columns: fix bug where vreplication of update statements were failing #7640
    • VReplication: Make the frequency at which heartbeats update the _vt.vreplication table configurable #7659
    • VReplication: Error out if binlog compression is turned on #7670
    • VReplication: Tablet throttler: support for custom query & threshold #7541
    • VStream API: allow aligning streams from different shards to minimize skews across the streams #7626

    OnlineDDL

    • OnlineDDL: update gh-ost binary to v1.1.1 #7394
    • Online DDL via VReplication #7419
    • Online DDL: VReplicatoin based mini stress test CI #7492
    • OnlineDDL: Revert for VReplication based migrations #7478
    • Online DDL: Internal support for eta_seconds #7630
    • Online DDL: Support 'SHOW VITESS_MIGRATIONS' queries #7638
    • Online DDL: Support for REVERT VITESS_MIGRATION statement #7656
    • Online DDL: Declarative Online DDL #7725

    VTAdmin

    • VTAdmin: Add vtadmin-web build flag for configuring fetch credentials #7414
    • VTAdmin: Add cluster field to vtadmin-api's /api/gates response #7425
    • VTAdmin: Add /api/clusters endpoint to vtadmin-api #7426
    • VTAdmin: Add /api/schemas endpoint to vtadmin-api #7432
    • VTAdmin: [vtadmin-web] Add routes and simple tables for all entities #7440
    • VTAdmin: [vtadmin-web] Set document.title from route components #7450
    • VTAdmin: [vtadmin-web] Add DataTable component with URL pagination #7487
    • VTAdmin: [vtadmin-api] Add shards to /api/keyspaces response #7453
    • VTAdmin: [vtadmin-web] Add replaceQuery + pushQuery to useURLQuery hook #7507
    • VTAdmin: [vtadmin-web] An initial pass for tablet filters #7515
    • VTAdmin: [vtadmin-web] Add a Select component #7524
    • VTAdmin: [vtadmin-api] Add /vtexplain endpoint #7528
    • VTAdmin: [vtadmin-api] Reorganize tablet-related functions into vtadmin/cluster/cluster.go #7553
    • VTAdmin: Three small bugfixes in Tablets table around stable sort order, display type lookup, and filtering by type #7568
    • VTAdmin: [vtadmin] Add GetSchema endpoint #7596
    • VTAdmin: [vtadmin/testutil] Add testutil helper to manage the complexity of recursively calling WithTestServer #7601
    • VTAdmin: [vtadmin] Add FindSchema route #7610
    • VTAdmin: [vtadmin-web] Add simple /schema view with table definition #7615
    • VTAdmin: [vtadmin] vschemas api endpoints #7625
    • VTAdmin: [vtadmin] Add support for additional service healthchecks in grpcserver #7635
    • VTAdmin: [vtadmin] test refactors #7641
    • VTAdmin: [vtadmin] propagate error contexts #7642
    • VTAdmin: [vtadmin] tracing refactor #7649
    • VTAdmin: [vtadmin] GetWorkflow(s) endpoints #7662
    • VTAdmin: [vitessdriver|vtadmin] Support Ping in vitessdriver, use in vtadmin to healthcheck connections during Dial #7709
    • VTAdmin: [vtadmin] Add to local example #7699
    • VTAdmin: vtexplain lock #7724
    • VTAdmin: [vtadmin] Aggregate schema sizes #7751
    • VTAdmin: [vtadmin-web] Add comments + 'options' parameter to API hooks #7754
    • VTAdmin: [vtadmin-web] Add common max-width to infrastructure table views #7760
    • VTAdmin: [vtadmin-web] Add hooks + skeleton view for workflows #7762
    • VTAdmin: [vtadmin-web] Add a hasty filter input to the /schemas view #7779

    Other / Tools

    • [rulesctl] Implements CLI tool for rule management #7712

    Examples / Tutorials

    • Source correct shell script in README #7749

    Documentation

    • Add Severity Labels document #7542
    • Remove Google Groups references #7664
    • Move some commas around in README.md :) #7671
    • Add Andrew Mason to Maintainers List #7757

    Build/CI Environment Changes

    • Update java build versions to vitess 10.0.0 #7383
    • CI: check run analysis to post JSON from file #7386
    • Fix Dockerfiles for vtexplain and vtctlclient #7418
    • CI: Add descriptive names to vrep shards. Update test generator script #7454
    • CI: adding 'go mod tidy' test #7461
    • Docker builds vitess/vtctlclient to install curl #7466
    • Add VT_BASE_VER to vtexplain/Dockerfile #7467
    • Enable -mysql_server_version in vttestserver, and utilize it in vttestserver container images #7474
    • [vtctld | tests only] testtmclient refactor #7518
    • CI: skip some tests on forked repos #7527
    • Workflow to check make sizegen #7535
    • Add mysqlctl docker image #7557
    • Restore CI workflow shard 26, accidentally dropped #7569
    • Update CODEOWNERS #7586
    • CI: ci-workflow-gen turn string to array to reduce conflicts #7582
    • Add percona-toolkit (for pt-osc/pt-online-schema-change) to the docker/lite images #7603
    • CI: Use ubuntu-18.04 in tests #7614
    • [vttestserver] Fix to work with sharded keyspaces #7617
    • Add tools.go #7517
    • Make vttestserver compatible with persistent data directories #7718
    • Add vtorc binary for rpm,deb builds #7750
    • Fixes bug that prevents creation of logs directory #7761
    • [Java] Guava update to 31.1.1 #7764

    Functionality Neutral Changes

    • VTGate: Remove unused key.Destination.IsUnique() #7565
    • VTGate: Add information_schema query on prepare statement #7746
    • VTGate: Tests for numeric_precision and numeric_scale columns in information_schema #7763
    • Disable flaky test until it can be fixed #7623
    • Tests: reset stat at the beginning of test #7644
    • Cleanup mysql server_test #7645
    • vttablet: fix flaky tests #7543
    • Removed unused tests for Wordpress installation #7516
    • Fix unit test fail after merge #7550
    • Add test with NULL input values for vindexes that did not have any. #7552
    Source code(tar.gz)
    Source code(zip)
    vitess-10.0.0-rc1-29a494f.tar.gz(371.02 MB)
    vitess-10.0.0_rc1-29a494f.x86_64.rpm(368.74 MB)
    vitess_10.0.0-rc1-29a494f_amd64.deb(371.03 MB)
  • v9.0.0(Jan 26, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Incompatible Changes

    The following PRs made changes to behaviors that clients might rely on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.

    • The update to golang 1.15 (#7204) might break systems that use TLS certificates with a common name. A fix is documented here (https://github.com/golang/go/issues/40748#issuecomment-673612108)

    Vitess 9.0 is not compatible with the previous release of the Vitess Kubernetes Operator (2.2.0). A new version of the Operator (2.3.0) is available that is compatible.

    Known Issue(s)

    • VReplication errors when a fixed-length binary column is used as the sharding key #8080

    Bugs Fixed

    VTGate / MySQL compatibility

    • Set Global #6957
    • Set udv allow more expressions #6964
    • Bug which caused the connection to not close in case of error writing an error packet #6977
    • Bug fix #7048 for SelectNone in StreamExecute route engine #7050

    Other

    • Binary PK: fix bug where padding of binary columns was being done incorrectly #6963
    • Pad non-fractional part of DECIMAL type #6967
    • Bug fix regression in /healthz #7090
    • Fix metadata related operation hangs when zk down #7228
    • Fix accidentally-broken legacy vtctl output format #7285
    • Healthcheck: use isIncluded correctly to fix replica/rdonly routing bug #6904

    Functionality Added or Changed

    VTGate / MySQL compatibility

    • VTGate: Allow INSERT with all defaults #6969
    • VTGate: Allow YEAR column type with length specified #6975
    • VTGate: Retry Execute when reserved connection is lost #6983
    • VTGate: Use ephemeral buffer when reading rows #6990
    • VTGate: Allow time_zone in reserved connection #6998
    • VTGate: Improved support for UNION #7007
    • VTGate: Add DDL parser support for FULLTEXT indexes. #7001
    • VTGate: Extend comparability of EvalResult to support hash codes #7016
    • VTGate: Use proto equal method #7017
    • VTGate: Fix COM_STMT_EXECUTE packet decode #7020
    • VTGate: Adds Planning and Parsing Support for Create Index of MySQL 5.7 #7024
    • VTGate: UNION DISTINCT support on vtgate #7029
    • VTGate: Improve database ddl plan #7034
    • VTGate: Support for hex & shard in vindex query #7044
    • VTGate: Use distinct primitive to solve more queries #7047
    • VTGate: Route using vindex for composite IN clause #7049
    • VTGate: Optimise struct field layout #7052
    • VTGate: Refactoring of plan building #7054
    • VTGate: Rewrite joins written with the USING construct #6660
    • VTGate: Add option to GetSchema to only send the row count and data length over the wire #6985
    • VTGate: Adds Planning and Parsing Support for Create Database of MySQL 5.7 #7068
    • VTGate: Make sure to check all GROUP BY columns #7080
    • VTGate: Separate sub-query and derived table into different structs #7081
    • VTGate: Adds Planning and Parsing Support for Alter Database of MySQL 5.7 #7086
    • VTGate: Convert usages of DDL struct to DDLStatement interface #7096
    • VTGate: Adds Planning and Parsing Support for Drop Database of MySQL 5.7 #7098
    • VTGate: Restore SHOW SCHEMAS support; fixes #7100 #7102
    • VTGate: Refactor Code to create a separate struct for CREATE TABLE #7116
    • VTGate: Allows for vttestserver and vtcombo to respond to VtGateExecute. #7121
    • VTGate: Support for lock and unlock tables #7139
    • VTGate: Merge SelectDBA routes when possible #7140
    • VTGate: Adds support for all the rails queries using information_schema #7143
    • VTGate: Add support for unary expression in order by column #7163
    • VTGate: Skip query rewriting for dual table #7164
    • VTGate: Refactor Code to create a separate struct for ALTER VSCHEMA #7173
    • VTGate: Refactor Show plans #7185
    • VTGate: Show privilege support #7194
    • VTGate: Planning and Parsing Support for Drop Table, Drop View and Alter View #7178
    • VTGate: Cache only dml and select plans #7196
    • VTGate: Planning and Parsing Support for Alter Table #7199
    • VTGate: Add FindAllShardsInKeyspace to vtctldserver #7201
    • VTGate: Initial implementation of vtctld service #7128
    • VTGate: improve-log: FAILED_PRECONDITION #7215
    • VTGate: Default to false for system settings to be changed per session at the database connection level #7299
    • VTGate: Route INFORMATION_SCHEMA queries #6932
    • VTGate: Adds Planning and Parsing Support for Create Index of MySQL 5.7 #7024
    • VTGate: Log sql which exceeds max memory rows #7055
    • VTGate: Enable Client Session Tracking feature in mysql protocol #6783
    • VTGate: Show columns from table_name targeted like select queries #6825
    • VTGate: This PR adds logic to simplify subquery expressions that are simple to
    • VTGate: Adding MySQL Check Constraints #6865
    • VTGate: Manage read your own writes system settings #6871
    • VTGate: Allow table_schema comparisons #6887
    • VTGate: Additional options support for SELECT INTO and LOAD DATA #6872
    • VTGate: Fixes vtgate which throws an error in case of empty statements #6947
    • VTGate: [Forward Port] #6940 - Fix error handling in olap mode #6949
    • VTGate: Adds Planning and Parsing Support for Create View of MySQL 5.7 #7060
    • VTGate: fix error: cannot run Select on table "dual" #7118
    • VTGate: Allow system table to be set as default database #7150
    • VTGate: Move auto_increment from reserved to non reserved keyword #7162
    • VTGate: Add only expr of aliasedExpr to weightstring function #7165
    • VTGate: [9.0] don't try to compare varchars in vtgate #7271
    • VTGate: Load Data From S3 #6823
    • VTGate: Unnest simple subqueries #6831
    • VTGate: Adding MySQL Check Constraints #6869
    • VTExplain: Add sequence table support for vtexplain #7186
    • VSchema: Support back-quoted names #7073
    • Healthcheck: healthy list should be recomputed when a tablet is removed #7176
    • Healthcheck: Hellcatlk wants to merge 1 commit into master from master #6953

    Set Statement Support

    Set statement support has been added in Vitess. There are some system variables which are disabled by default and can be enabled using flag -enable_system_settings on VTGate. These system variables are set on the mysql server. Because they change the mysql session, using them leads to the Vitess connection no longer using the connection pool and forcing dedicated connections.

    VReplication

    • VReplication: refactored and enhanced support for JSON columns #6829
    • VReplication: Don't update tx timestamp on heartbeat #6930
    • VReplication E2E Tests: Refactored tests for readability and attempting to fix flakiness #6991
    • VRepl/Tablet Picker: improve observability of selected tablet #6999
    • VReplication: Handle comment statement type in vstreamer #7092
    • VReplication e2e: Fine tuned test to reduce flakiness and added more logging to debug future flakiness #7138
    • VReplication: Make relay log size & rows configurable. #6992
    • VReplication: New workflows cli UX. Allow reads/writes to be switched independently #7071
    • VReplication: DropSources: change table rename logic #7230
    • VReplication: MoveTables: delete routing rules and update vschema on Complete and Abort #7234
    • VReplication: V2 Workflow Start: wait for streams to start and report errors if any while starting a workflow #7248
    • VReplication: Ignore temp tables created by onlineddl #7159
    • VReplication: Set time zone to UTC while streaming rows #6845
    • VReplication: Materialization and character sets: Add test to verify/demo a workaround for charset issues while using string functions in filters #6847
    • VReplication: Tool to diagnose vreplication issues in production #6892
    • VReplication: Allow multiple blacklists for master #6816
    • VStreamer Field Event: add allowed values for set/enum #6981
    • VDiff: lock keyspace while snapshoting, restart target in case of errors #7012
    • VDiff: make enums comparable #6880
    • VDiff: add ability to limit number of rows to compare #6890
    • VDiff/Tablet Picker: fix issue where vdiff sleeps occasionally for tablet picker retry interval #6944
    • [vtctld]: fix error state in Workflow Show #6970
    • [vtctld] Workflow command: minor fixes #7008
    • MoveTables: validate that source tables exist, move all tables #7018
    • SwitchWrites bug: reverse replication workflows can have wrong start positions #7169

    VTTablet

    • VTTablet: fast and reliable state transitions #7011
    • VTTablet: don't shutdown on too many connections #7039
    • VTTablet: debug/env page to change variables in real-time #7189
    • VTTablet: Adds better errors when there are timeouts in resource pools #7002
    • VTTablet: Return to re-using server IDs for binlog connections #6941
    • VTTablet: Correctly initialize the TabletType stats #6989
    • Backup: Use pargzip instead of pgzip for compression. #7037
    • Backup: Add s3 server-side encryption and decryption with customer provided key #7088

    OnlineDDL

    • Online DDL: follow ups in multiple trajectories #6901
    • Online DDL: cancel running migrations executed by another tablet #7006
    • OnlineDDL: Adding ddl_strategy session variable #7042
    • Online DDL: ddl_strategy session variable and vtctl command line argument #7045
    • Online DDL: Removing online ddl query hint from ALTER TABLE #7069
    • Online DDL: vtgate -ddl-strategy flag renamed to -ddl_strategy #7074 Automatically retry migration that was interrupted during master failover Automatically terminate migrations run by a failed tablet
    • Online DDL:request_context/migration_context #7082
    • Online DDL: Support CREATE, DROP statements in ApplySchema and online DDL #7083
    • Online DDL: ddl_type column #7097
    • OnlineDDL: "cancel-all" command to cancel all pending migrations in keyspace #7099
    • OnlineDDL: Support vtctl OnlineDDL <keyspace> show <context> #7145
    • OnlineDDL: Normalizing Online-DDL queries #7153
    • Online DDL: ddl_strategy=direct #7172
    • Online DDL: Executor database pool size increase #7206
    • Online DDL: DROP TABLE translated to RENAME TABLE statement #7221
    • Online DDL: Adding @@session_uuid to vtgate; used as 'context' #7263
    • Online DDL: ignore errors if extracted gh-ost binary is identical to installed binary #6928
    • Online DDL: Table lifecycle: skip time hint for unspecified states #7151
    • Online DDL: Migration uses low priority throttling #6830
    • Online DDL: Fix parsing of online-ddl command line options #6900
    • OnlineDDL bugfix: make sure schema is applied on tablet #6910
    • OnlineDDL: request_context/migration_context #7082
    • OnlineDDL: Fix missed rename in onlineddl_test #7148
    • OnlineDDL: Online DDL endtoend tests to support MacOS #7168

    VTadmin

    • VTadmin: Initial vtadmin-api, clusters, and service discovery #7187
    • VTadmin: The tiniest possible first implementation of vtadmin-web #7218
    • VTadmin: Add cluster protos to discovery and vtsql package constructors #7224
    • VTadmin: Add static file service discovery implementation #7229
    • VTadmin: Query vtadmin-api from vtadmin-web with fetch + react-query #7239
    • VTadmin: Move allow_alias option in MySqlFlag enum to precede the aliased IDs #7166
    • [vtctld]: vtctldclient generator #7238

    Other

    • Fix comment typo #6974
    • Fix all occurrences of fmt.Sprint(x) where x is int #7244
    • Fix incorrect comments #7257
    • Fix comment for IDPool #7212
    • IsInternalOperationTableName: see if a table is used internally by vitess #7104
    • Add timeout for mysqld_shutdown #6849
    • Should receive healthcheck updates from all tablets in cells_to_watch #6852
    • Workflow listall with no workflows was missing newline #6853
    • Allow incomplete SNAPSHOT keyspaces #6863

    Examples / Tutorials

    • Examples/operator: fix tags and add vtorc example #7358
    • local docker: copy examples/common into /vt/common to match MoveTables user guide #7252
    • Update docker-compose examples to take advantage of improvements in Vitess #7009

    Documentation

    • Vitess Slack Guidelines v1.0 #6961
    • Do vschema_customer_sharded.json before create_customer_sharded.sql #7210
    • Added readme for the demo example #7226
    • Adding @shlomi-noach to CODEOWNERS #6855
    • Add Rohit Nayak to maintainers #6903
    • 7.0.3 Release Notes #6902
    • 8_0_0 Release Notes #6958
    • Update maintainers of Vitess #7093
    • Updating Email Address #7095
    • Update morgo changes #7105
    • Move PR template to .github directory #7126
    • Fix trivial typo #7179
    • Add @ajm188 + @doeg to CODEOWNERS for vtctld service files #7202
    • Add @ajm188 + @doeg as vtadmin codeowners #7223:w

    Build Environment Changes

    • Clean up plan building test cases #7057
    • Fix unit test error #6953, #6993
    • Fixing the 5.6 builds of vitess/lite #6960
    • Pin mariadb to use mariadb-server-10.2 #6966
    • Replace vitess:base with vitess:lite images for docker-compose services #7004
    • Fix flakey TestParallelRunnerApprovalFirstRunningSecondRunning test #7014
    • Allow custom image tags in compose #7043
    • Support statsd for vitess #7072
    • Add vtctl to make install-local #7125
    • Updating Java unit tests for JDK9+ compatibility #7144
    • Add Go Version to Bootstrap Image #7182
    • Update Vitess v8.0 images #7174
    • Fix broken package ref in UBI docker build #7183
    • Convert CentOS extra packages installation to yum instead of downloading #7188
    • Make docker_local: fix missing mysql_server package #7213
    • Add unit test case to improve test coverage for go/sqltypes/result.go #7227
    • Update Golang to 1.15 #7204
    • Add linter configuration #7247
    • Modify targets to restore behavior of make install #6842
    • Download zookeeper 3.4.14 from archive site #6865
    • Bump junit from 4.12 to 4.13.1 in /java #6870
    • Fix ListBackups for gcp and az to work with root directory #6873
    • Pulling bootstrap resources from vitess-resources #6875
    • [Java] Bump SNAPSHOT version to 9.0 after Vitess release 8.0 #6907
    • Change dependencies for lite builds #6933
    • Truncate logged query in dbconn.go. #6959
    • [GO] go mod tidy #7137
    • goimport proto files correctly #7264
    • Cherry pick version of #7233 for release-9.0 #7265
    • Update Java version to 9.0 #7369
    • Adding curl as dependency #6965

    Functionality Neutral Changes

    • Healthcheck: add unit test for multi-cell replica configurations #6978
    • Healthcheck: Correct Health Check for Non-Serving Types #6908
    • Adds timeout to checking for tablets. #7106
    • Remove deprecated vtctl commands, flags and vttablet rpcs #7115
    • Fixes comment to mention the existence of reference tables. #7122
    • Updated pull request template to add more clarity #7193
    • Redact password #7198
    • action_repository: no need for http.Request #7124
    • Testing version upgrade/downgrade path from/to 8.0 #7323
    • Use context from Go's standard library #7235
    • Update operator.yaml backup engine description #6832
    • Docker - upgrade to Debian Buster #6833
    • Updating azblob to remove directory after removing backup #6836
    • Fixing some flaky tests #6874
    • Flaky test: attempt to fix TestConnection in go/test/endtoend/messaging #6879
    • Stabilize test #6882
    • Tablet streaming health fix: never silently skip health state changes #6885
    • Add owners to /go/mysql #6886
    • Fixes a bug in Load From statement #6911
    • Query consolidator: fix to ignore leading margin comments #6917
    • Updates to Contacts section as Reporting #7023
    • Create pull_request_template #7027
    • Fixed pull request template path #7062

    Backport

    • Backport: [vtctld] Fix accidentally-broken legacy vtctl output format #7292
    • Backport #7276: Vreplication V2 Workflows: rename Abort to Cancel #7339
    • Backport #7297: VStreamer Events: remove preceding zeroes from decimals in Row Events
    • Backport #7255: VReplication DryRun: Report current dry run results for v2 commands #7345
    • Backport #7275: VReplication: Miscellaneous improvements #7349
    • Backport 7342: Workflow Show: use timeUpdated to calculate vreplication lag #7354
    • Backport 7361: vtctl: Add missing err checks for VReplication v2 #7363
    • Backport 7297: VStreamer Events: remove preceding zeroes from decimals in Row Events #7340
    Source code(tar.gz)
    Source code(zip)
    vitess-9.0.0-daa6085.tar.gz(318.07 MB)
    vitess-9.0.0-daa6085.x86_64.rpm(316.22 MB)
    vitess_9.0.0-daa6085_amd64.deb(318.08 MB)
  • v9.0.0-rc1(Jan 7, 2021)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Incompatible Changes

    The following PRs made changes to behaviors that clients might rely on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.

    • The update to golang 1.15 might break systems that use TLS certificates with a common name. A fix is documented here (https://github.com/golang/go/issues/40748#issuecomment-673612108)

    Bugs Fixed

    • Set Global #6957
    • Binary PK: fix bug where padding of binary columns was being done incorrectly #6963
    • Set udv allow more expressions #6964
    • Pad non-fractional part of DECIMAL type #6967
    • Bug which caused the connection to not close in case of error writing an error packet #6976
    • Bug which caused the connection to not close in case of error writing an error packet #6977
    • Bug fix #7048 for SelectNone in StreamExecute route engine #7050
    • Bug fix regression in /healthz #7090
    • Fix metadata related operation hang when zk down #7228

    Functionality Added or Changed

    • Vtctld: fix error state in Workflow Show #6970
    • Adds better error when there are timeouts in resource pools #7002
    • Backup: Use pargzip instead of pgzip for compression. #7037
    • Vschema: Support back-quoted names #7073
    • IsInternalOperationTableName: see if a table is used internally by vitess #7104
    • Log sql which exceeds max memory rows #7055
    • Add sequence table support for vtexplain #7186
    • Vtctld: vtctldclient generator #7238

    VTGate / MySQL compatibility

    • VTGate: Allow INSERT with all defaults #6969
    • VTGate: Allow YEAR column type with length specified #6975
    • VTGate: Retry Execute when reserved connection is lost #6983
    • VTGate: Use ephemeral buffer when reading rows #6990
    • VTGate: Allow time_zone in reserved connection #6998
    • VTGate: Improved support for UNION #7007
    • VTGate: Add DDL parser support for FULLTEXT indexes. #7001
    • VTGate: Extend comparability of EvalResult to support hash codes #7016
    • VTGate: Use proto equal method #7017
    • VTGate: Fix COM_STMT_EXECUTE packet decode #7020
    • VTGate: Adds Planning and Parsing Support for Create Index of MySQL 5.7 #7024
    • VTGate: UNION DISTINCT support on vtgate #7029
    • VTGate: Improve database ddl plan #7034
    • VTGate: Support for hex & shard in vindex query #7044
    • VTGate: Use distinct primitive to solve more queries #7047
    • VTGate: Route using vindex for composite IN clause #7049
    • VTGate: Optimise struct field layout #7052
    • VTGate: Refactoring of plan building #7054
    • VTGate: Rewrite joins written with the USING construct #6660
    • VTGate: Add option to GetSchema to only send the row count and data length over the wire #6985
    • VTGate: Adds Planning and Parsing Support for Create Database of MySQL 5.7 #7068
    • VTGate: Make sure to check all GROUP BY columns #7080
    • VTGate: Separate sub-query and derived table into different structs #7081
    • VTGate: Adds Planning and Parsing Support for Alter Database of MySQL 5.7 #7086
    • VTGate: Convert usages of DDL struct to DDLStatement interface #7096
    • VTGate: Adds Planning and Parsing Support for Drop Database of MySQL 5.7 #7098
    • VTGate: Restore SHOW SCHEMAS support; fixes #7100 #7102
    • VTGate: Refactor Code to create a separate struct for CREATE TABLE #7116
    • VTGate: Allows for vttestserver and vtcombo to respond to VtGateExecute. #7121
    • VTGate: Support for lock and unlock tables #7139
    • VTGate: Merge SelectDBA routes when possible #7140
    • VTGate: Adds support for all the rails queries using information_schema #7143
    • VTGate: Add support for unary expression in order by column #7163
    • VTGate: Skip query rewriting for dual table #7164
    • VTGate: Refactor Code to create a separate struct for ALTER VSCHEMA #7173
    • VTGate: Refactor Show plans #7185
    • VTGate: Show privilege support #7194
    • VTGate: Planning and Parsing Support for Drop Table, Drop View and Alter View #7178
    • VTGate: Cache only dml and select plans #7196
    • VTGate: Planning and Parsing Support for Alter Table #7199
    • VTGate: Add FindAllShardsInKeyspace to vtctldserver #7201
    • VTGate: improve-log: FAILED_PRECONDITION #7215

    VReplication

    • VReplication: refactored and enhanced support for JSON columns #6829
    • VReplication: Don't update tx timestamp on heartbeat #6930
    • VReplication E2E Tests: Refactored tests for readability and attempting to fix flakiness #6991
    • VRepl/Tablet Picker: improve observability of selected tablet #6999
    • VReplication: Handle comment statement type in vstreamer #7092
    • VReplication e2e: Fine tuned test to reduce flakiness and added more logging to debug future flakiness #7138
    • VReplication: Make relay log size & rows configurable. #6992
    • VReplication: New workflows cli UX. Allow reads/writes to be switched independently #7071
    • VReplication: DropSources: change table rename logic #7230
    • VReplication: MoveTables: delete routing rules and update vschema on Complete and Abort #7234
    • VReplication: V2 Workflow Start: wait for streams to start and report errors if any while starting a workflow #7248

    VTtablet

    • VTtablet: ignore errors if extracted gh-ost binary is identical to installed binary #6928
    • VTtablet: fast and reliable state transitions #7011
    • VTtablet: don't shutdown on too many connections #7039
    • VTtablet: debug/env page to change variables in real-time #7189

    VStreamer

    • VStreamer Field Event: add allowed values for set/enum #6981

    VDiff

    • VDiff: lock keyspace while snapshoting, restart target in case of errors #7012

    OnlineDDL

    • Online DDL: follow ups in multiple trajectories #6901
    • Online DDL: cancel running migrations executed by another tablet #7006
    • OnlineDDL: Adding ddl_strategy session variable #7042
    • Online DDL: ddl_strategy session variable and vtctl command line argument #7045
    • Online DDL: Removing online ddl query hint from ALTER TABLE #7069
    • Online DDL: vtgate -ddl-strategy flag renamed to -ddl_strategy #7074 Automatically retry migration that was interrupted during master failover Automatically terminate migrations run by a failed tablet
    • Online DDL:request_context/migration_context #7082
    • Online DDL: Support CREATE, DROP statements in ApplySchema and online DDL #7083
    • Online DDL: ddl_type column #7097
    • OnlineDDL: "cancel-all" command to cancel all pending migrations in keyspace #7099
    • OnlineDDL: Support vtctl OnlineDDL <keyspace> show <context> #7145
    • OnlineDDL: Normalizing Online-DDL queries #7153
    • Online DDL: ddl_strategy=direct #7172
    • Online DDL: Executor database pool size increase #7206
    • Online DDL: DROP TABLE translated to RENAME TABLE statement #7221
    • Online DDL: Adding @@session_uuid to vtgate; used as 'context' #7263

    VTadmin

    • VTadmin: Initial vtadmin-api, clusters, and service discovery #7187
    • VTadmin: The tiniest possible first implementation of vtadmin-web #7218
    • VTadmin: Add cluster protos to discovery and vtsql package constructors #7224
    • VTadmin: Add static file service discovery implementation #7229
    • VTadmin: Query vtadmin-api from vtadmin-web with fetch + react-query #7239

    Other

    • Route INFORMATION_SCHEMA queries #6932
    • Return to re-using server IDs for binlog connections #6941
    • Fix comment typo #6974
    • Correctly initialize the TabletType stats #6989
    • Clean up plan building test cases #7057
    • Add s3 server-side encryption and decryption with customer provided key #7088
    • Fix all occurrences of fmt.Sprint(x) where x is int #7244
    • local docker: copy examples/common into /vt/common to match MoveTables user guide #7252
    • Fix incorrect comments #7257

    Examples / Tutorials

    Documentation

    • Vitess Slack Guidelines v1.0 #6961
    • Update docker-compose examples to take advantage of improvements in Vitess #7009
    • Adds Planning and Parsing Support for Create Index of MySQL 5.7 #7024
    • Do vschema_customer_sharded.json before create_customer_sharded.sql #7210
    • Fix comment for IDPool #7212
    • Added readme for the demo example #7226

    Build Environment Changes

    • Fix unit test error #6953, #6993
    • Fixing the 5.6 builds of vitess/lite #6960
    • Set udv allow more expressions #6964
    • Pin mariadb to use mariadb-server-10.2 #6966
    • Replace vitess:base with vitess:lite images for docker-compose services #7004
    • Fix flakey TestParallelRunnerApprovalFirstRunningSecondRunning test #7014
    • Allow custom image tags in compose #7043
    • Support statsd for vitess #7072
    • Add vtctl to make install-local #7125
    • Updating Java unit tests for JDK9+ compatibility #7144
    • Add Go Version to Bootstrap Image #7182
    • Update Vitess v8.0 images #7174
    • Fix broken package ref in UBI docker build #7183
    • Convert CentOS extra packages installation to yum instead of downloading #7188
    • Make docker_local: fix missing mysql_server package #7213
    • Add unit test case to improve test coverage for go/sqltypes/result.go #7227
    • Use context from Go's standard library #7235
    • Update Golang to 1.15 #7204
    • Add linter configuration #7247

    Functionality Neutral Changes

    • Healthcheck: add unit test for multi-cell replica configurations #6978
    • Healthcheck: healthy list should be recomputed when a tablet is removed #7176
    • Vtctld Workflow command: minor fixes #7008
    • Adds timeout to checking for tablets. #7106
    • Remove deprecated vtctl commands, flags and vttablet rpcs #7115
    • Fixes comment to mention the existence of reference tables. #7122
    • Updated pull request template to add more clarity #7193
    • Redact password #7198
    • action_repository: no need for http.Request #7124
    • Table lifecycle: skip time hint for unspecified states #7151
    Source code(tar.gz)
    Source code(zip)
    vitess-9.0.0-rc1-0472d47.tar.gz(317.61 MB)
    vitess-9.0.0_rc1-0472d47.x86_64.rpm(315.75 MB)
    vitess_9.0.0-rc1-0472d47_amd64.deb(317.61 MB)
  • v8.0.0(Oct 27, 2020)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Incompatible Changes

    The following PRs made changes to behavior that clients might be relying on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.

    • Change error code from 1203 to 1153 for three specific error conditions. #6630 The three errors being changed: When a GRPC message coming back to the vtgate exceeds the configured size limit (configured via grpc_max_message_size) If a write is attempting to make a change with a payload that is larger than the configured allowable size If a scatter query is attempting to collect and process too many rows in memory within the vtgate to construct a response
    • Zero auto-increment mode: This enables inserting a 0 value (and not just NULL) into an auto-increment column to generate sequence values. This matches the default behavior of MySQL. #6749
    • Turn off schema tracker by default #6712

    Known Issue(s)

    • VReplication errors when a fixed-length binary column is used as the sharding key #8080

    Bugs Fixed

    VTGate / MySQL compatibility

    • Fix where clause in information schema with correct database name #6599
    • Fix DDL execution on reserved connection without in active transaction #6514
    • Fix allow passed table_schema in information_schema queries to vttablet if the keyspace is not found #6653
    • Fix information schema for non-existent schema #6667
    • Fix reserved connection in autocommit mode on DML #6748
    • Fix dbname that needs escaping that we broke in fce4cfd894 #6794
    • MySQL CLI, mysql --ssl does not work in MySQL 8.0 #5556
    • OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') #6208
    • Fix support for show tables where tables_in_db to use keyspace name, not underlying db name #6446
    • Common empty result set query on MySQL returns a row in Vitess #6663
    • Operator precedence issue #6734
    • Table names out-of-order between replica and primary #6738
    • Close Idle reserved connections and Rollback Idle transactions #6552
    • Minor Vindex fixes: #6526
    • VTgate does not report autocommit system variable and SERVER_STATUS_IN_TRANS flag correctly #5825
    • MariaDB JDBC Connector fails to connect to vtgate with default options #5851
    • VTgate panics if there are no vttablets and vtctlds #6117
    • Remove shard session held in vtgate for reserved connection on connection failure #6522
    • Session variable does not work as expected in 7.0 #6559
    • Retry should not happen when in dedicated connection #6523
    • Correctly report AUTOCOMMIT status in network packets #6551
    • Close Idle reserved connections and Rollback Idle transactions #6552
    • Reset Session for Reserved Connection Query Failure #6673
    • Errors in multi-statement queries not fatal to statement sequence #6747 #6808
    • Lock wait timeout on DELETE from table using lookup index #6698
    • Application couldn't connect to external db giving error tabletserver.go:1522] Code: INVALID_ARGUMENT syntax error at position 33 near ':vtg2' (CallerID: gnocchi) : Sql: "select convert(:vtg1, CHAR(:vtg2)) as anon_1 from dual", #6466
    • The client got the wrong database name #6753
    • DBAPIError exception wrapped from (pymysql.err.InternalError) (1105, u'vtgate: http://gnocchi-gnocchizone-vtgate:15000/: reserved connections are not supported on old gen gateway') #6577
    • Provide support for “SELECT INTO OUTFILE S3” #6811
    • Fix error around breaking of multi statements #6824
    • Support SHOW TABLE STATUS FROM db type queries in vtgate #6354
    • SHOW FULL FIELDS doesn't follow routing rules #6803
    • VDiff/Tablet Picker: fix picker retry logic to handle canceled context #6954
    • Healthcheck: Use isIncluded correctly to fix replica/rdonly routing bug #6922

    Other

    • VReplication tablet_picker should keep trying to find a tablet until context expires #6546
    • VReplication improve reverse workflow: update cells/tablet_types #6556
    • VDiff: fix panic for tables with a unicode_loose_md5 vindex #6640
    • Vtctld UI: Add mysqld port display back to vtctld web UI #6598 #6597
    • Vtctld: Fix workflow CopyState fetching (and thus printing). #6657
    • Fix unsaved event regression #6710
    • Change tracker to use allprivs instead of dba #6720
    • Fixes long wait filter keyspace #6721
    • s3 ListBackups doesn’t work from root directory #6732
    • Vtctld: ExecuteFetchAsDba and ExecuteFetchAsAllPrivs do not escape db names. #6545
    • Can't switch on restore_from_backup on existing shard with no tables #4896
    • Refactor conn.go #6818

    Functionality Added or Changed

    VTGate / MySQL compatibility

    • Add support for DELETE IGNORE #6722
    • Only take on simple dual queries in the vtgate #6666
    • Support mycli access to vtgate #4365
    • Add support for unicode_loose_xxhash #6457
    • Provide virtual information_schema #5394
    • information_schema.tables_schema comparison with dynamic value no longer works #6827
    • Mysqldump: Allow setting multiple variables in one statement #5401
    • Block lock function usage other than dual table #6470
    • Lock Session Support #6517
    • Make sure lookup vindexes are queryable inside transaction #6499
    • Add unicode_loose_xxhash Vindex type #6549
    • Support vindex update or delete when destination is provided by client in dml query #6554
    • Make sure to handle EXPLAIN on tablets #6579
    • Rewrite SHOW TABLES #6615
    • Make offset work in OLAP mode #6655
    • Support sql_calc_found_rows with limit in sharded keyspace #6680
    • Lock shard session heartbeat #6683
    • Mysql -ssl tag is deprecated in the later versions of mysql #6682
    • Use statement support in olap mode #6692
    • Delete row using consistent lookup vindex in where clause #6700
    • Add vtexplain support for non-evenly sharded keyspaces #6705
    • Make sure dual queries set found_rows correctly #6718
    • Add if not exists support to database ddl in parsing #6724
    • Allow Update of same vindex using same vindex lookup on consistent lookup vindex #6736
    • Update Vindex only on changes #6737
    • Add support for SHOW (DATABASES|VITESS_SHARDS|VITESS_TABLETS) LIKE #6750
    • 'CASCADE' support #6743
    • Mask database name #6791
    • Make sure to backtick schema names properly #6550
    • Transaction Limiter: fix quota leak if Begin() errors or if the tx killer runs #6731
    • Allow Load Data From S3 #6823

    OLAP Functioanlity

    • Allow switching between OLAP and OLTP #6691
    • Use statement support in olap mode #6692
    • Fix error handling in olap mode #6940

    Set Statement Support

    Set statement support is added in Vitess. There are some system variables which are disabled by default and can be enabled using flag -enable_system_settings on VTGate.These system variables are set on the backing MySQL instance, and will force the connection to be dedicated instead of part of the connection pool.

    • Disabled passthrough system variables by default. #6859
    • Allow switching workload between OLAP and OLTP #4086 #6691
    • Support for multiple session set statement syntax #6508 #6495 #6494
    • Allow enumeration in set system variables #6493
    • Handle boolean settings better #6501
    • Set statements refactored to use plan building #6487
    • Add more system settings #6486
    • Evaluate system variables #6708

    VReplication

    • VReplication: _vt.vreplication source column VARBINARY->BLOB, allowing more tables to participate in a workflow #6421
    • VReplication and vstreamer performance and error metrics #6519, #6132
    • VReplication: Find collations for char primary keys and cast comparison during catchup #6568
    • Varchar primary keys with collations can cause vreplication to miss rows #6622
    • VReplication: Timestamp w/ Heartbeat #6635
    • VReplication: Improve few error/log messages #6669
    • VReplication: handle SET statements in the binlog stream #6772
    • VReplication: e2e Tests added to CI #6730
    • VDiff logs: additional logging around vdiff/vstreamer for observability #6684
    • VStream: experimental POC of mysql protocol #6670
    • Materialize: Support rollups using a literal value as a column #6733
    • Materialization: Ignore non-participating tables that are renamed/deleted #6778
    • Materialize: add ability for maintaining accurate table-level aggregates #6767
    • Schema Tracker: Turn off schema tracker by default #6712 (breaking change)
    • Workflow: Add List-All Command #6533
    • Workflow: List -> Show, and Expand Metadata #6544
    • Refactor: Cleanup Noisy Workflow Logging #6744
    • Schema tracker: Change tracker to use the allprivs user instead of dba #6720
    • vtctl: Adding VExec and Workflow commands #6410
    • vtctl Workflow cmd: don't expect all shards to have workflow streams #6576
    • Reshard: Sort table definitions before comparing them #6765 #6738
    • Expose a shard range calculator as a vtctl command GenerateShardRanges #6751 #6752
    • VStream: Field event now has all column attributes #6525
    • VStream events do not populate field "Flags" on FIELD events #5914
    • RFC: VReplication based SplitClone #4604
    • Experimental: Mysql Protocol support for VStream #6675
    • VReplication: Add metrics around vstreamer and vreplication metrics #6787
    • Fix issues with missing tables in vstreams #6789
    • Add validations and logs to fix common issues faced during MoveTables/Reshard #6814
    • Allow multiple blacklists for master #6816

    VTtablet

    • VTtablet throttling #6668 https://vitess.io/docs/reference/features/tablet-throttler/
    • TabletManager: publish displayState #6648
    • TabletManager: call setReadOnly inside ChangeTabletType #6762
    • TabletManager: change how SetReadOnly is called to avoid errors from externally managed tablets #6786
    • TabletManager: call SetReadOnly inside ChangeTabletType #6804
    • vttablet: Open and Close healthStreamer #6515
    • vttablet: create database if not present #6490
    • Restore: do not change tablet type to RESTORE if not actually performing a restore #6679
    • Restore: checkNoDB should not require tables to be present #6695
    • Heartbeat: Additional metrics/stats (lag histogram) #6528 #6634
    • VTtablet throttler feature flag: -enable-lag-throttler #6815
    • VTtablet two-phase commit design doc #6498

    VTorc - Orchestrator

    Following PRs are experimental version of Vitess-native Orchestrator 'vtorc' is ready for users to try out.

    • Orchestrator: initial import of Orchestrator #6582
    • Orchestrator: vitess mode #6613
    • Orchestrator: more changes #6714
    • Orchestrator: use contexts with timeout for remote operations #6780
    • Orchestrator: fixed orchestrator govet errors #6781
    • Orchestrator: Add clusters_to_watch flag to. Defaults to all clusters. #6821

    Point in Time Recovery

    • PITR: Fix deadlock in ff code #6774
    • Add flags to allow PITR binlog client TLS support #6775

    Other

    • Add realtime health stats to vtctld /api/keyspace/ks/tablets API #6569
    • vtctld UI: Fix logic for displaying vindexes #6603
    • Online schema changes #6547
    • Table lifecycle management #6719, docs
    • Online-DDL: migration uses low priority throttling #6837
    • Emergency Reparent Refactor #6449
      • Make emergency reparents more robust. #6206
    • Reparent test failures #6706
    • Update user-facing terminology in the vtctld2 UI #6481
    • GetSchema: Batch/parallel access to underlying mysqld for lower latency #6491
    • Prometheus interface: vtgate_buffer_* metrics cardinality #6326
    • Show more vindex details in vtctld /app/schema UI #6450
    • GRPC: update enforcement policy on server to match the one from the client #6629

    Examples / Tutorials

    • region_sharding: working resharding example #6565
    • vtcompose/docker-compose: fix InitShardMaster #6584
    • [docker] fix tablet hostname #6509

    Documentation

    • Replace more uses of slave with replica/replication #6520
    • Update some option docstrings for accuracy. #6590
    • Minor addition to Materialize help text. #6627
    • Remove auto-generation comment #6728
    • add skip flag that can skip comparing source & destination schema when run splitdiff #6477
    • Fix and clarify some help options #6817

    Build Environment Changes

    • Address #6631 by reducing optimization level #6632
    • Add front-end build tooling + fix build error #6473
    • Replace gogo proto with golang #6571
    • Improve make proto #6580
    • Remove chromium hard-coding #6636
    • Docker - allow BUILD_NUMBER variable override #6628
    • Docker/lite: Fix package URLs for ubi images. #6704
    • Helm & docker/k8s update #6723
    • Fix build errors in ./go/vt/vtgate for go1.15 #6800
    • Fix build error on vcursor_impl tests on newer versions of go #6799
    • [java] bump java version to 8.0.0-SNAPSHOT for next release #6478
    • Helm/docker: fix Docker builds + tag 7.0.2 #6773
    • Docker: Upgrade to Debian Buster (release-8.0) #6888
    • Docker: Pin mysql 5.7 and 8.0 versions for xtrabackup compatibility
    • Docker: Revert "Docker - upgrade to Debian Buster (release-8.0)" #6929
    • Zookeper: Download zookeeper 3.4.14 from archive site #6867

    Functionality Neutral Changes

    • Routing cleanup: remove routeOptions #6531
    • PITR: testcase #6594
    • Replaced Error with a warning in case parsing of VT_SPAN_CONTEXT fails #6766
    • Reparent tests refactoring: setup/teardown for each test, reduce cut/paste, improve readability #6726
    • Decider: endtoend test infrastructure + tests #6770
    • Orchestrator: add more test cases #6801
    • Orchestrator: don't set DbName for tests that don't need it. #6812
    • Add unit test case to improve test coverage for go/sqltypes/named_result.go #6672
    • End to end: deflake sharding tests #6715
    • Workflow test: fix flaky test. remove obsolete code #6694
    • Add unit test for func Proto3ValuesEqual. #6649
    • Ensure tests for discoverygateway #6536
    • Convert syslog events to normal log events #6511
    • Add diagnostic logging to healthcheck. #6512
    • Healthcheck should receive healthcheck updates from all tablets in cells_to_watch #6852
    • Named columns and convenience SQL query interface #6543
    • Check http response code in vtctld API tests #6570
    • Reverting package-lock.json to the one pre-PR #6603 #6611
    • Removed sqlparser.preview to set logstats stmtType and use plan.type #6637
    • AST struct name rewording #6642
    • Fix: Remove SetMaster Query Expectation from ERS Test #6617
    • Fix path for vttablet query log in end to end test #6656
    • Fixes error log #6496
    • Add diagnostic logging to healthcheck. #6535
    • Try to install apt keys from list of different keyservers #6674
    • Trivial copy-pasta comment fixup #6592
    • Remove long-unused memcache and cacheservice #6596
    • Fix MoveTables docstring; was not valid JSON. #6606
    • Tablet gateway: unit tests #6608
    • Tablet gateway doesn't see all tablets in cells from cells_to_watch #6846
    • Add Sleep to ERS Unit Test #6610
    • Strings to enum #6729
    • Remove syslog dependency #3563
    • Operator precedence must take associativity into consideration #6758
    • Add optional ignore to delete statement #6802
    • Invalid trace payload causes vitess to fail the request and return an error #6759
    • Vttablet: minor logging fix #6564
    • Vttablet: add more logging to checkMastership #6618
    Source code(tar.gz)
    Source code(zip)
    vitess-8.0.0-7e09d0c.tar.gz(328.59 MB)
    vitess-8.0.0-7e09d0c.x86_64.rpm(326.65 MB)
    vitess_8.0.0-7e09d0c_amd64.deb(328.60 MB)
  • v7.0.3(Oct 22, 2020)

    This update fixes several regressions that were deemed significant enough to be backported to the release branch.

    Bugs Fixed

    • Vtgate : Should receive healthcheck updates from all tablets in cells_to_watch #6857
    • Vtgate : Fix error around breaking of multistatements #6824
    • Vtgate : Fix bug in REPLICA/RDONLY routing #6904
    • Vttablet : checkNoDB should not require tables to be present #6788

    Known Issues

    This release includes MySQL 5.7.32 and Percona Xtrabackup 2.4.20 which have a known incompatibility. Percona is working on a resolution.

    The issue will manifest if a user using 5.7.32:
    * Creates a new compressed table.
    * Runs any ALTER TABLE  that uses the algorithm copy (table rebuild) on a compressed table.
    

    Build Changes

    • Build : Download zookeeper 3.4.14 from archive site #6868
    Source code(tar.gz)
    Source code(zip)
    vitess-7.0.3-5f29393.tar.gz(321.68 MB)
    vitess-7.0.3-5f29393.x86_64.rpm(319.81 MB)
    vitess_7.0.3-5f29393_amd64.deb(321.70 MB)
  • v8.0.0-rc1(Oct 7, 2020)

    This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components.

    Incompatible Changes

    The following PRs made changes to behavior that clients might be relying on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment.

    • Change error code from 1203 to 1153 for three specific error conditions. #6630 The three errors being changed: When a GRPC message coming back to the vtgate exceeds the configured size limit (configured via grpc_max_message_size) If a write is attempting to make a change with a payload that is larger than the configured allowable size If a scatter query is attempting to collect and process too many rows in memory within the vtgate to construct a response
    • Zero auto-increment mode: This enables inserting a 0 value (and not just NULL) into an auto-increment column to generate sequence values. This matches the default behavior of MySQL. #6749
    • Turn off schema tracker by default #6712

    Bugs Fixed

    VTGate / MySQL compatibility

    • Fix where clause in information schema with correct database name #6599
    • Fix DDL execution on reserved connection without in active transaction #6514
    • Fix allow passed table_schema in information_schema queries to vttablet if the keyspace is not found #6653
    • Fix information schema for non-existent schema #6667
    • Fix reserved connection in autocommit mode on DML #6748
    • Fix dbname that needs escaping that we broke in fce4cfd894 #6794
    • MySQL CLI, mysql --ssl does not work in MySQL 8.0 #5556
    • OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') #6208
    • Fix support for show tables where tables_in_db to use keyspace name, not underlying db name #6446
    • Common empty result set query on MySQL returns a row in Vitess #6663
    • Operator precedence issue #6734
    • Table names out-of-order between replica and primary #6738
    • Close Idle reserved connections and Rollback Idle transactions #6552
    • Minor Vindex fixes: #6526
    • VTgate does not report autocommit system variable and SERVER_STATUS_IN_TRANS flag correctly #5825
    • MariaDB JDBC Connector fails to connect to vtgate with default options #5851
    • VTgate panics if there are no vttablets and vtctlds #6117
    • Remove shard session held in vtgate for reserved connection on connection failure #6522
    • Session variable does not work as expected in 7.0 #6559
    • Retry should not happen when in dedicated connection #6523
    • Correctly report AUTOCOMMIT status in network packets #6551
    • Close Idle reserved connections and Rollback Idle transactions #6552
    • Reset Session for Reserved Connection Query Failure #6673
    • Errors in multi-statement queries not fatal to statement sequence #6747 #6808
    • Lock wait timeout on DELETE from table using lookup index #6698
    • Application couldn't connect to external db giving error tabletserver.go:1522] Code: INVALID_ARGUMENT syntax error at position 33 near ':vtg2' (CallerID: gnocchi) : Sql: "select convert(:vtg1, CHAR(:vtg2)) as anon_1 from dual", #6466
    • The client got the wrong database name #6753
    • DBAPIError exception wrapped from (pymysql.err.InternalError) (1105, u'vtgate: http://gnocchi-gnocchizone-vtgate:15000/: reserved connections are not supported on old gen gateway') #6577
    • Provide support for “SELECT INTO OUTFILE S3” #6811

    Other

    • VReplication tablet_picker should keep trying to find a tablet until context expires #6546
    • VReplication improve reverse workflow: update cells/tablet_types #6556
    • VDiff: fix panic for tables with a unicode_loose_md5 vindex #6640
    • Vtctld UI: Add mysqld port display back to vtctld web UI #6598 #6597
    • Vtctld: Fix workflow CopyState fetching (and thus printing). #6657
    • Fix unsaved event regression #6710
    • Change tracker to use allprivs instead of dba #6720
    • Fixes long wait filter keyspace #6721
    • s3 ListBackups doesn’t work from root directory #6732
    • Vtctld: ExecuteFetchAsDba and ExecuteFetchAsAllPrivs do not escape db names. #6545
    • Can't switch on restore_from_backup on existing shard with no tables #4896
    • Refactor conn.go #6818

    Functionality Added or Changed

    VTGate / MySQL compatibility

    • New functionality to set workload = 'olap'; can not repeat, but set workload = 'oltp' can; #4086

    • Add support for DELETE IGNORE #6722

    • Only take on simple dual queries in the vtgate #6666

    • Support mycli access to vtgate #4365

    • Add support for unicode_loose_xxhash #6457

    • Provide virtual information_schema #5394

    • Mysqldump: Allow setting multiple variables in one statement #5401

    • SET planning #6487

    • Allow enumeration in set system variables #6493

    • Allow set to have scope for each setting #6494

    • HandleSet in executor to handle multiple settings in single set statement #6495

    • Passthrough system settings flag: enabled by default #6586

    • Handle boolean settings better #6501

    • Add more system settings #6486

    • Block lock function usage other than dual table #6470

    • Lock Session Support #6517

    • Make sure lookup vindexes are queryable inside transaction #6499

    • Add unicode_loose_xxhash Vindex type #6549

    • Support vindex update or delete when destination is provided by client in dml query #6554

    • Make sure to handle EXPLAIN on tablets #6579

    • Rewrite SHOW TABLES #6615

    • Make offset work in OLAP mode #6655

    • Support sql_calc_found_rows with limit in sharded keyspace #6680

    • Lock shard session heartbeat #6683

    • Mysql -ssl tag is deprecated in the later versions of mysql #6682

    • Allow switching between OLAP and OLTP #6691

    • Use statement support in olap mode #6692

    • Delete row using consistent lookup vindex in where clause #6700

    • Add vtexplain support for non-evenly sharded keyspaces #6705

    • Evaluate system variables #6708

    • Make sure dual queries set found_rows correctly #6718

    • Add if not exists support to database ddl in parsing #6724

    • Allow Update of same vindex using same vindex lookup on consistent lookup vindex #6736

    • Update Vindex only on changes #6737

    • Add support for SHOW (DATABASES|VITESS_SHARDS|VITESS_TABLETS) LIKE #6750

    • Support multiple session set statement syntax #6508

    • 'CASCADE' support #6743

    • Mask database name #6791

    • Make sure to backtick schema names properly #6550

    • Transaction Limiter: fix quota leak if Begin() errors or if the tx killer runs #6731

    OLAP Functioanlity

    • Allow switching between OLAP and OLTP #6691
    • Use statement support in olap mode #6692

    System Variables

    • Allow enumeration in set system variables #6493
    • Allow set to have scope for each setting #6494
    • HandleSet in executor to handle multiple settings in single set statement #6495

    VReplication

    • VReplication: _vt.vreplication source column VARBINARY->BLOB, allowing more tables to participate in a workflow #6421
    • VReplication and vstreamer performance and error metrics #6519, #6132
    • VReplication: Find collations for char primary keys and cast comparison during catchup #6568
    • Varchar primary keys with collations can cause vreplication to miss rows #6622
    • VReplication: Timestamp w/ Heartbeat #6635
    • VReplication: Improve few error/log messages #6669
    • VReplication: handle SET statements in the binlog stream #6772
    • VReplication: e2e Tests added to CI #6730
    • VDiff logs: additional logging around vdiff/vstreamer for observability #6684
    • VStream: experimental POC of mysql protocol #6670
    • Materialize: Support rollups using a literal value as a column #6733
    • Materialization: Ignore non-participating tables that are renamed/deleted #6778
    • Materialize: add ability for maintaining accurate table-level aggregates #6767
    • Schema Tracker: Turn off schema tracker by default #6712 (breaking change)
    • Workflow: Add List-All Command #6533
    • Workflow: List -> Show, and Expand Metadata #6544
    • Refactor: Cleanup Noisy Workflow Logging #6744
    • Schema tracker: Change tracker to use the allprivs user instead of dba #6720
    • vtctl: Adding VExec and Workflow commands #6410
    • vtctl Workflow cmd: don't expect all shards to have workflow streams #6576
    • Reshard: Sort table definitions before comparing them #6765 #6738
    • Expose a shard range calculator as a vtctl command GenerateShardRanges #6751 #6752
    • VStream: Field event now has all column attributes #6525
    • VStream events do not populate field "Flags" on FIELD events #5914
    • RFC: VReplication based SplitClone #4604
    • Experimental: Mysql Protocol support for VStream #6675
    • VReplication: Add metrics around vstreamer and vreplication metrics #6787
    • Fix issues with missing tables in vstreams #6789
    • Add validations and logs to fix common issues faced during MoveTables/Reshard #6814

    VTtablet

    • VTtablet throttling #6668
    • TabletManager: publish displayState #6648
    • TabletManager: call setReadOnly inside ChangeTabletType #6762
    • TabletManager: change how SetReadOnly is called to avoid errors from externally managed tablets #6786
    • TabletManager: call SetReadOnly inside ChangeTabletType #6804
    • vttablet: Open and Close healthStreamer #6515
    • vttablet: create database if not present #6490
    • Restore: do not change tablet type to RESTORE if not actually performing a restore #6679
    • Restore: checkNoDB should not require tables to be present #6695
    • Heartbeat: Additional metrics/stats (lag histogram) #6528 #6634
    • VTtablet throttler feature flag: -enable-lag-throttler #6815

    VTorc - Orchestrator

    • Orchestrator: initial import of Orchestrator #6582
    • Orchestrator: vitess mode #6613
    • Orchestrator: more changes #6714
    • Orchestrator: use contexts with timeout for remote operations #6780
    • Orchestrator: fixed orchestrator govet errors #6781
    • Orchestrator: Add clusters_to_watch flag to. Defaults to all clusters. #6821

    Point in Time Recovery

    • PITR: Fix deadlock in ff code #6774
    • Add flags to allow PITR binlog client TLS support #6775

    Other

    • Add realtime health stats to vtctld /api/keyspace/ks/tablets API #6569
    • vtctld UI: Fix logic for displaying vindexes #6603
    • Online schema changes #6547
      • Table lifecycle management #6719, docs: https://vitess.io/docs/reference/features/table-lifecycle/
    • Emergency Reparent Refactor #6449
      • Make emergency reparents more robust. #6206
    • Update user-facing terminology in the vtctld2 UI #6481
    • GetSchema: Batch/parallel access to underlying mysqld for lower latency #6491
    • Prometheus interface: vtgate_buffer_* metrics cardinality #6326
    • Show more vindex details in vtctld /app/schema UI #6450
    • GRPC: update enforcement policy on server to match the one from the client #6629

    Examples / Tutorials

    • region_sharding: working resharding example #6565
    • vtcompose/docker-compose: fix InitShardMaster #6584
    • [docker] fix tablet hostname #6509

    Documentation

    • Replace more uses of slave with replica/replication #6520
    • Update some option docstrings for accuracy. #6590
    • Minor addition to Materialize help text. #6627
    • Remove auto-generation comment #6728
    • Add skip flag that can skip comparing source & destination schema when run splitdiff #6477
    • Fix and clarify some help options #6817

    Build Environment Changes

    • Address #6631 by reducing optimization level #6632
    • Add front-end build tooling + fix build error #6473
    • Replace gogo proto with golang #6571
    • Improve make proto #6580
    • Remove chromium hard-coding #6636
    • Docker - allow BUILD_NUMBER variable override #6628
    • Docker/lite: Fix package URLs for ubi images. #6704
    • Helm & docker/k8s update #6723
    • Fix build errors in ./go/vt/vtgate for go1.15 #6800
    • Fix build error on vcursor_impl tests on newer versions of go #6799
    • [java] bump java version to 8.0.0-SNAPSHOT for next release #6478
    • Helm/docker: fix Docker builds + tag 7.0.2 #6773

    Functionality Neutral Changes

    • Routing cleanup: remove routeOptions #6531
    • PITR: testcase #6594
    • Replaced Error with a warning in case parsing of VT_SPAN_CONTEXT fails #6766
    • Reparent tests refactoring: setup/teardown for each test, reduce cut/paste, improve readability #6726
    • Decider: endtoend test infrastructure + tests #6770
    • Orchestrator: add more test cases #6801
    • Orchestrator: don't set DbName for tests that don't need it. #6812
    • Add unit test case to improve test coverage for go/sqltypes/named_result.go #6672
    • End to end: deflake sharding tests #6715
    • Workflow test: fix flaky test. remove obsolete code #6694
    • Add unit test for func Proto3ValuesEqual. #6649
    • Ensure tests for discoverygateway #6536
    • Convert syslog events to normal log events #6511
    • Add diagnostic logging to healthcheck. #6512
    • Named columns and convenience SQL query interface #6543
    • Check http response code in vtctld API tests #6570
    • Reverting package-lock.json to the one pre-PR #6603 #6611
    • Removed sqlparser.preview to set logstats stmtType and use plan.type #6637
    • AST struct name rewording #6642
    • Fix: Remove SetMaster Query Expectation from ERS Test #6617
    • Fix path for vttablet query log in end to end test #6656
    • Fixes error log #6496
    • Add diagnostic logging to healthcheck. #6535
    • Try to install apt keys from list of different keyservers #6674
    • Trivial copy-pasta comment fixup #6592
    • Remove long-unused memcache and cacheservice #6596
    • Fix MoveTables docstring; was not valid JSON. #6606
    • Tablet gateway: unit tests #6608
    • Add Sleep to ERS Unit Test #6610
    • Strings to enum #6729
    • Remove syslog dependency #3563
    • Operator precedence must take associativity into consideration #6758
    • Add optional ignore to delete statement #6802
    • Invalid trace payload causes vitess to fail the request and return an error #6759
    • Vttablet: minor logging fix #6564
    • Vttablet: add more logging to checkMastership #6618
    Source code(tar.gz)
    Source code(zip)
    vitess-8.0.0-rc1-0988455.tar.gz(328.52 MB)
    vitess-8.0.0_rc1-0988455.x86_64.rpm(326.59 MB)
    vitess_8.0.0-rc1-0988455_amd64.deb(328.52 MB)
  • v7.0.2(Sep 22, 2020)

    This update fixes several regressions that were deemed significant enough to be backported to the release branch.

    Bugs Fixed

    • vtgate : Operator precedence must take associativity into consideration #6764
    • vtgate: Fix reserved connection in autocommit mode on DML #6748
    • VDiff: fix panic for tables with a unicode_loose_md5 vindex #6745
    • vttablet: Turn off schema tracker by default #6746
    • restore: do not change tablet type to RESTORE if not actually performing a restore #6687
    • vttablet : Tablet manager init should publish displayState #6686
    • vttablet: Tablet manager add more logging to checkMastership #6685
    • vttablet : Fix replmanager deadlock #6634
    Source code(tar.gz)
    Source code(zip)
    vitess-7.0.2-aea21dc.tar.gz(321.68 MB)
    vitess-7.0.2-aea21dc.x86_64.rpm(319.80 MB)
    vitess_7.0.2-aea21dc_amd64.deb(321.69 MB)
  • v7.0.1(Aug 25, 2020)

    This update fixes several regressions that were deemed significant enough to be backported to the release branch. In addition, the experimental feature to set system variables is now behind a flag that disables it by default.

    Configuration Changes

    • Vtgate: added flag enable_system_settings to enable passthrough system settings, defaults to false #6562

    Bugs Fixed

    • Vreplication: Materialize should only get schema from source tablets if target is missing tables #6623
    • Vtctld UI: mysql port is not displayed in tablet list #6597 #6600
    • Vtgate: transaction killer was closing reserved connection after the specified time. Only idle connections should be closed. Idle transactions should be rolled back. #6583
    • Vtcompose: docker-compose example was broken by tabletmanager changes #6587
    • Vreplication: tablet_picker should respect canceled context. Also, keep trying to find a tablet until context expires. #6567
    • Vtgate: vindex update/delete by destination #6541 #6572
    • Vtgate: some usages of EXPLAIN were broken #6574 #6581
    • Vtgate: handle multiple settings in single SET statement #6504
    • Vtgate: cleanup shard session for reserved connection on failure #6563
    • Vtgate: if executing DDL on reserved connection, start transaction if needed #6555
    • Vtgate: make sure lookup vindexes use same transaction if one is open #6505
    • Healthcheck: add diagnostic logging, do not add tablet if it is shutting down #6512
    Source code(tar.gz)
    Source code(zip)
    vitess-7.0.1-19c92a5.tar.gz(321.63 MB)
    vitess-7.0.1-19c92a5.x86_64.rpm(319.75 MB)
    vitess_7.0.1-19c92a5_amd64.deb(321.63 MB)
  • v6.0.20-20200818(Aug 18, 2020)

  • v7.0.0(Jul 28, 2020)

    Incompatible Changes

    This release includes the following changes which may result in incompatibilities when upgrading from a previous release. It is important that Vitess components are upgraded in the recommended order. This will change in the next release as documented in VEP-3.

    • VTGate: The default gateway (-gateway_implementation flag value) is now "tabletgateway". The old default was “discoverygateway”. To use the old gateway, specify the flag in the command line. Anyone already using the flag will see no change; to use the new gateway, remove the -gateway_implementation flag. The new gateway is simpler and uses a new healthcheck module with no loss of functionality. #6155

    • VTGate: VTGate incorrectly returned Warning Code field name as Type. This has now been fixed. Code or scripts that depend on the incorrect behavior might break. #6437

    • VReplication: The "-cell" parameter to MoveTables has been renamed to “-cells”, since it now accepts multiple comma-separated cells or cellsAliases. The parameter specifies which cells can provide tablets as the source for vreplication streams. #6456

    • VTGate: The possibility of sending OpenTracing trace spans through query comments has been changed. The old way of doing it was severely broken, so our assumption is that no-one has successfully been using this feature. Still, this is an API change, and we are reporting it as a breaking change. #6463

    • VTTablet has been refactored to be generally more responsive to state changes and its environment: #6131 #6139 #6241 #6263 #6311 #6345 #6348 #6384 #6396 #6432 #6461

      • Some initialization parameters are now mandatory: init_keyspace, init_shard and init_tablet_type. The flow where you could issue a vtctlclient InitTablet and then invoke vttablet without those init parameters is no longer supported.

      • Consequently vtctl's InitTablet and UpdateTabletAddrs are deprecated.

      • demote_master_type is deprecated: Now that init_tablet_type is mandatory, that value can act as the demote_master_type. There is no need for a separate flag.

      • VTTablet will publish the discovered MySQL port on startup, and will not update it during the life of the process. There is no known use case where a MySQL port is expected to change. So, there is no need to keep polling MySQL for its port.

      • Decouple enable_replication_watcher from disable_active_reparents. Currently, vttablet attempts to fix replication only if enable_replication_watcher is true and disable_active_reparents is false. This behavior has changed to depend only on the disable_active_reparents flag.

      • Transitioning from a serving type to SPARE used to go through a brief lameduck period specified by the serving_state_grace_period (default 0). This has been removed. The transition will now happen immediately even if a grace period is specified.

      • It was possible to enable health reporting and heartbeat at the same time. Performing the two at the same time is not useful. As of this release, enabling heartbeat automatically disables the polling mode of checking for replica health.

      • The ACL rules used to be reported on the vttablet status page. The design was using an incorrect framework. This has been removed.

    • Build: make targets "build_web" and “embed_static” have been removed. Instead there are new targets “web_bootstrap”, “web_build” and “web_start” #6473

    Deprecations

    • The following vtctl commands have been deprecated. Deprecated commands will continue to work until the next release (8.0). #6428 #6345 #6461

      • StartSlave -> Replaced with StartReplication

      • StopSlave -> Replaced with StopReplication

      • ChangeSlaveType -> Replaced with ChangeTabletType

      • InitTablet

      • UpdateTabletAddrs

    • Various vtctl commands had a flag called "-wait_slave_timeout". It has been replaced with “-wait_replicas_timeout”. Scripts using the old flag will continue to work until the next major release (8.0). #6428

    • Helm charts: The current helm charts have not been keeping up with the forward progress in vitess, and there has been no maintainer for it. Additionally, the operator has pulled ahead of what the helm charts could do. The operator can also be functionally more versatile than the helm charts because custom code can be written to manage complex workflows. To maintain focus and velocity in the project, we are deprecating the helm charts. #6439

    • Consul: The usage of Consul has been problematic for vitess users. The main difficulty comes from the fact that it's non-trivial for a multi-cell deployment. Due to this, we want to discourage the use of consul as a topo server, and are deprecating support for it. #6440

    Bugs Fixed

    VTGate / MySQL compatibility

    • Handle non-keyword tokens FORMAT, TREE and VITESS. (This was broken by #6110) #6445

    • When using user-defined variables that have not been initialized first, a non-fatal panic occurs and the connection is dropped. #6294 #6308

    • Setting system variable panics vtgate. #6162 #6163

    • regression: USE does not work from MySQL client. #6153 #6157

    • USE does not return an error if a keyspace doesn't exist. #3060 #6157

    • Dotted table names are not escaped properly in lookup vindex flow. #6142 #6145

    • Produce correct error code (1105 instead of 1062) when inserting duplicates into consistent_lookup vindex. #6144

    • Rewriting expressions fail when used as function parameters. #6281

    • Rollback after error clobbers updates. #6285

    • Queries should not be wrapped into transactions if autocommit=1 is set. #6264

    • Prepared statements with bind variables in column specification produce ‘missing bind var’ error. #6287 #6298

    • Prepared statement queries returning NULL results are not parsed. #6240 #6249

    • DEFAULT function not working with status column. #6221 #6224

    • VTGate panic kills connection when selecting database() when using OLAP workload. #6200 #6205

    • Wrong column value can be inserted due to parameterization variable confusion. #6266 #6270

    • Fix owned table name formatting and duplicate error code in consistent lookup. #6145

    • Support for case-insensitive user defined variables. #6225

    • Fix performance regression on VTGate. #6203

    • Return NULL for non-existent user-defined variables. #6308

    • MySQL connection closing should not inflate query counts. #6226

    • show vitess_shards intermittently returns empty list. #5038 #5189

    • Protect insert lookup queries from bind variable name clashes #6328

    • CAST/CONVERT should work correctly #6304

    • Allow charset introducers for vindex columns #6301

    • Fix panic from EXPLAIN #6204

    • Handle special comments that start with ! #6186

    • Fix panic with uppercase types #6140

    • Allow ‘$’ in identifiers #6136

    Other

    • VTTablet: Etcd query timeout because auto-committable transactions are not properly rolled back. #6258 #6299

    • VTTablet: Fixed race in schema engine during notification of schema changes. #6268

    • VTTablet: Reloading ACL config was throwing misleading error messages.his has now been fixed and the error messages improved. #6309

    • VTTablet: Panic during backup caused an error in the pgzip library has been fixed by contributing a fix to pgzip and upgrading to a new release that includes the fix. #6189

    • VTTablet: Improvement of error handling by retrying while uploading to S3. #6182 #6395 #6405

    • VTTablet: builtinbackupengine was uploading a manifest despite errors from S3. Error collection during backup has been fixed to address this. #6349 #6350

    • VTTablet: Fix dba connection leak in LockTables. #6424

    • VTTablet: Fix connection pool leak in schema engine. #6426

    • VTTablet: Fix regression where a heartbeat interval flag setting of 1s would have resulted in an internal value of 1000s #6356 #6357

    • VTTablet: Fix panic from binlog_streamer when ‘-watch_replication_stream’ is set., #6253

    • VTTablet: Add context to MysqlDaemon schema functions. This lets us handle disconnected RPCs instead of the client hanging forever. #6243

    • VTTablet: fix panic from txlogz #6238

    • VTTablet: Add back support for millisecond values for flags that was dropped when yaml config was introduced #6250

    • Vtctld UI: Change all external links in vitess to use https #6170 #6169

    • Ensure SrvVSchema gets rebuilt for all cells #6276

    • Fix MariadbGTIDSet multi-domain support #6184

    • mysqlctl: Fix connection leak during killConnection() #6245

    • Fix release script to install examples in the correct directory #6159

    • examples/operator: fix port-forward command #6418

    • VStream client: should send current position events immediately #6391 #6397

    Functionality Added or Changed

    VTGate/ MySQL compatibility

    • Transactions on read-only replicas are now supported. Start a transaction by first choosing the desired tablet_type ("use [email protected]"), then issuing a “BEGIN” statement. Internally, this will issue a “START TRANSACTION READ ONLY” on the underlying database. This is only available if using the new gateway_implementation of tabletgateway (which is now the default). #6166 #6244

    • Create better plans for NULL comparisons. Instead of using a SelectScatter, now vtgate will produce empty results when the predicate is a comparison against literal NULLs. #6133 #6425

    • Support IN, NOT IN and equality (=) with null values #6152 #6146

    • Add EXPLAIN support for vtgate plans. View the produced vtgate query plan by using explain format=vitess #6110

    • Allow users to perform savepoint related constructs. #6355 #6404 #6412 #6414

    • Added support for user-level advisory lock functions like get_lock and release_lock. This is considered experimental and will be improved in future releases based on user feedback. #6367 #6370 #6470

    • Allow users to change system variable settings at session level using set statements. This is a major change that was merged later in the development cycle. It is considered experimental and there may be edge cases. #6107 #6459 #6472 #6488

    • "ignore_nulls" option to lookup vindexes #6147 #6222: There are situations where the "from" columns of a lookup vindex can be null. Such columns cannot be inserted in the lookup due to the uniqueness constraints of a lookup. The new “ignore_nulls” option of the lookup vindex allows you to accommodate null values in rows. Such rows can still be found through other means, like the primary vindex.

    • Add support for row_count() function. #6174 #6179

    • XOR Operator is now supported. #6369 #6371

    • Allow 'binary' for character_set_results. It is equivalent to null. #6237

    • Improve logging by adding keyspace, table, and tablet_type. #6282

    • VTGate now takes in optional flags for ‘max_payload_size’ and ‘warn_payload_size’. A payload size above the warning threshold will only produce a warning. A payload size that exceeds ‘max_payload_size’ will result in an error. The default behavior is not to check payload size, which is backwards-compatible. #6143 #6375

    • A new comment directive (‘/*vt+ IGNORE_MAX_PAYLOAD_SIZE=1 */) has been added to provide the ability to override the payload size checking per query. #6143

    • A new comment directive (‘/*vt+ IGNORE_MAX_MEMORY_ROWS=1 */) has been added to provide the ability to ignore the "-max_memory_rows" setting per query. #6430

    • Support ‘select * from information_schema.processlist’. #6310

    • Support batch lookup queries for integer and binary types. Other types will continue to require one query per row. #6420

    • Support system settings at session level. #6149

    • Prepare for future features by adding reserved connections. This is the ability for a connected client to have a dedicated connection instead of using the connection pool. #6303 #6313 #6402 #6441 #6451

    • Allow charset introducers with vindex column values. #6236 #6314

    • Support Union All between different shards. #6351

    • Exclude KeyspaceId(s) from plan cache key.This improves the cache hit rate. #6233

    • Reload TLS config on SIGHUP. #6215

    • Drop vindex DDL. #6151

    • show vitess_tablets now reports master_term_start_time. We ensure that this is reported correctly after reparents. #6135 #6292 #6293

    • Allow parenthesized selects #6342

    • Support DO statements #6239

    • Support calling database() inside other functions #6291

    • Support IN, NOT IN and equality (=) with null values #6152 #6146

    VReplication

    • VStream Copy extends the VStream API to allow streaming of the contents of databases or tables. Earlier you had to specify a position to start streaming events from. #6277

    • VReplication workflows now accept "-cells" as an option. This can be a comma-separated list of cells or cellsAliases. The source tablet for vreplication will be chosen randomly among the tablets that belong to the specified cells/cellsAliases. If the option is not provided, we default to the cell that is local to the vreplication target. #6442, #6456 #6465

    • Schema Versioning. Tablets now have a schema tracker that can be enabled to store schema versions as DDLs are applied. The _vt.schema_version table contains the schema with the corresponding ddl and gtid position when the ddl was applied. The tracked schemas are used by the new Historian service to provide vstreams with the correct schema while streaming old events. #6300 #6164 #6278 #6399 #6411 #6435 #6448

    • Support for specifying multiple external sources to facilitate migration into vitess.

    • Foreign Keys are now supported on the target database during vreplication. In the copy phase we disable FK checks and re-enable them once the replication phase starts. #6284

    • Materialize: added a drop_constraint option during table copy to strip constraints while creating target tables #6422

    • Materialize: optimizations while creating target tables resulting in a significant improvement in performance when moving a large number of tables from an external MySQL instance #6207, #6251

    • DropSources: Added ‘-rename_tables’ option to rename the source tables rather than dropping them entirely. #6383

    • Savepoints: Vreplication streams are not able to handle savepoint events in the binlog. #6438

    • Add keyspace, shard and workflow labels to VReplicationSecondsBehindMaster metrics #6279

    • Correct command line help for MigrateServedTypes, MigrateServedFrom and SwitchWrites #6187

    • VExec and Workflow commands have been added to vtctld as a layer above VReplicationExec. They allow listing/stopping/starting/deleting workflows and ability to run custom queries on the _vt.vreplication table for all shards in a workflow. #6410

    Point-in-time Recovery

    In the 4.0 release we introduced functionality to recover a keyspace into a snapshot based on a specified time. That feature only allowed you to restore a backup, but did not have the ability to automatically replay binlogs up to the specified time. The feature has now been completed so that true point-in-time recovery is possible. This is currently only supported for MySQL compatible installations, not MariaDB. #4857 #6267 #6408

    Healthcheck

    • VTGate can now use a new healthcheck that is logically simpler while providing the same functionality. This is the default. The older healthcheck has been deprecated but is still available to use if desired by setting "-gateway_implementation=discoverygateway" #5750 #6155

    • VTCombo has been migrated to use the new healthcheck #6302

    • VTExplain has been migrated to use new healthcheck #6460

    MiniVitess

    A docker container that can be used to bring up a self-contained vitess cluster, which can automatically resolve and self-configure to meet an external MySQL cluster. While this might sound very similar to VTCombo and vttestserver, it is not quite the same.

    MiniVitess spins up in docker. Given a MySQL server hostname, schema name & credentials, it runs an automated topology analysis via orchestrator, identifying the cluster members. It proceeds to bring up distinct tablet for each server and opens up vtgate connections. #6377

    Examples / Tutorials

    • Vitess operator example #6154

    • Region sharding example #6209 #6275

    • Change local example to use new command DropSources #6178

    • Are-you-alive example now makes it easier to test multiple endpoints #6202

    • Local docker tutorial added to website

    • Simplified helm chart for examples (however, helm is now deprecated) #6168

    Other

    • vtctlclient ListAllTablets now shows the MasterTermStartTime. This is shown as "" for read-only tablet types. In case of an old master that has not yet updated the topo record, TabletType will be reported as “unknown” #6135

    • New debug handlers have been added to enable/disable golang block and mutex profiling. #6137

    • Vitessdriver: Allow overriding driver name. Default stays as "vitess" #6138

    • VTCombo: add options to allow starting MySQL within VTCombo. Note that vttestserver provides similar functionality. #6265

    • Disable foreign key checks during preflight schema change tests #4696

    • Fix labels for opentsdb #6289

    • Allow empty shards when running ValidateSchemaKeyspace behind flag ‘-skip-no-master’ #6216

    • Death by SIGPIPE on stdout/stderr can be avoided by setting the new flag ‘catch-sigpipe’ #6297

    • Prepared Statement support added when using special vitess construct of use <keyspace>:<shard> by adding GetField support in engine plan execute #6132

    • GTIDSet Union #6180

    • Add relay log position and file based positions to ReplicationStatus #6217

    • Function that can find errant GTIDs #6296

    • VTTablet: Add StopReplicationMode to StopReplicationAndGetStatus RPC #6335

    • VTTablet: DemoteMaster RPC returns full status instead of string representation of current position #6365

    • VTTablet: WaitForPosition RPC can now accept either file-based or GTID-based positions #6374

    Documentation

    • Fix links to local example and sample client in vitessdriver godoc #6254

    • Flag documentation has been improved #6261

    • All flags are now documented in the program reference on the website.

    Build Environment Changes

    • golangci-lint now runs as a pre-commit hook and CI check #6385 #6406

    • CI test for region_sharding example #6275

    • Removed staticcheck as a pre-commit hook because it is included in golangci-lint #6416

    • Faster unit tests #6415

    • Consul-api version upgraded to 1.8.0 #6358

    • Upgrade pgzip to v1.2.4 #6189

    • Upgrade log4j2 version to 2.13.3 #6403

    • Upgrade netty version to 4.1.48.Final #6317

    • Upgrade tcnative version to 2.0.30.Final #6317

    • Add docker_local target to Makefile and a docker/local/run.sh script #6393

    • Only check for k3s on Linux #6353

    • UI build tooling was broken, it has now been fixed. #6473

    Functionality Neutral Changes

    • Refactor table filtering logic #6242 #6259

    • VTGate: Replace Sprintf with string concatenation in normalizer, this reduces CPU usage on benchmarks #6127

    • VTTablet: Reduce the number of packets sent from vttablet to MySQL #6130

    • Fixed flakiness in External Connector Test: fixed race #6201

    • Fixed flakiness in Message Test: test had not been updated after jitter functionality was added #6283

    • Fix flakiness in TestConnectTimeout #6188

    • Fix TLS test for Go 1.13+ #6185

    • Fixed linter errors #6416 #6364 #6307

    • More dollar-sign tests #6363

    • VTExplain: use DiscoveryGateway (and old healthcheck) until migration can be completed #6248

    • Deprecated RPCs from 6.0 have been deleted #6380

    • Update references to orchestrator repo to point to openark instead of github #6360

    • Terminology: deprecate or replace references to "slave" with “replica” except where referring to actual sql commands or fields returned from MySQL. #6428 #6392 #6379

    • Terminology: replace user-facing references to "slave" with “replica” or “replication” in UI #6481

    • Fix log format - use Infof instead of Info #6413

    • Cleanup temp files used in tests #6400

    • Add unit test for func stringMapToString #6280

    • Check errors in etcd2topo unit test #6128

    • Unit test for atomic.go #6120

    • Unit test for ResolveIPv4Addrs #6230

    • Enabled set statement tests #6167

    • Replaced VTTablet ExecuteBatch api call with Execute for autocommit query. #6407 #6431

    • Fixed flaky TestQueryPlanCache #6454

    • Fix test compile errors #6434

    • Scatter_conn unit test using new healthcheck #6458

    • Test that vtexplain works with healthcheck #6252

    • Tests: Removed error check that caused spurious test failures #6462

    • More test cases for IGNORE_MAX_PAYLOAD_SIZE #6467

    • VSCopy Test: Mods/bug fixes/test logging to make test repeatable/debuggable, fix flakiness #6341 #6235

    • Cleanup scatter conn #6257

    • Simplify vtgate executor #6256

    • Add tests for SHOW COLUMNS #6192

    • Added fuzzer #6175

    • Simplified TxPool and ConnPool code #6150

    Source code(tar.gz)
    Source code(zip)
    vitess-7.0.0-a3a5232.tar.gz(321.61 MB)
    vitess-7.0.0-a3a5232.x86_64.rpm(319.74 MB)
    vitess_7.0.0-a3a5232_amd64.deb(321.62 MB)
  • v7.0.0-beta(Jul 21, 2020)

    Incompatible Changes

    • VTGate: The default gateway ("-gateway_implementation" flag value) is now "tabletgateway". The old default was "discoverygateway”. To use the old gateway, specify the flag in the command line. Anyone already using the flag will see no change; to use the new gateway, remove the -gateway_implementation flag. The new gateway is simpler and uses a new healthcheck module with no loss of functionality. #6155

    • VTGate: VTGate incorrectly returned Warning Code field name as Type. This has now been fixed. Code or scripts that depend on the incorrect behavior might break. #6437

    Deprecations

    • The following vtctl commands have been deprecated. Deprecated commands will continue to work until the next release (8.0). #6428

    • StartSlave -> Replaced with StartReplication

    • StopSlave -> Replaced with StopReplication

    • ChangeSlaveType -> Replaced with ChangeTabletType

    • Various vtctl commands had a flag called "-wait_slave_timeout". It has been replaced with “-wait_replicas_timeout”. Scripts using the old flag will continue to work until the next major release (8.0). #6428

    • Helm chart (more details to be added)

    • Consul (more details to be added)

    • Replaced VTTablet ExecuteBatch api call with Execute for autocommit query. #6407 #6431

    Bugs Fixed

    VTGate / MySQL compatibility

    • VTGate: When using user-defined variables that have not been initialized first, a non-fatal panic occurs and the connection is dropped. #6294 #6308

    • VTGate: Setting system variable panics vtgate. #6162 #6163

    • regression: USE does not work from mysql client. #6153 #6157

    • USE does not return an error if a keyspace doesn't exist. #3060 #6157

    • Dotted table names are not escaped properly in lookup vindex flow. #6142 #6145

    • Error 1105 instead of 1062 when inserting duplicate into consistent_lookup vindex. #6144

    • VTGate: Rewriting expressions fail when used as function parameters. #6281

    • Rollback after error clobbers updates. #6285

    • Queries should not be wrapped into transactions if autocommit=1 is set. #6264

    • Prepared statements with bind variables in column specification produce ‘missing bind var’ error. #6287 #6298

    • Prepared statement queries returning NULL results are not parsed. #6240 #6249

    • DEFAULT function not working with status column. #6221 #6224

    • VTGate panic kills connection when selecting database() when using OLAP workload. #6200 #6205

    • Wrong column value can be inserted due to parameterization variable confusion. #6266 #6270

    • Fix owned table name formatting and duplicate error code in consistent lookup. #6145

    • Support for case-insensitive user defined variable. #6225

    • Fix performance regression on VTGate. #6203

    • Return NULL for non-existent user-defined variables. #6308

    • Lock function support. #6370

    • VTGate: MySQL connection closing should not inflate query counts. #6226

    • VTGate: Show vitess_shards intermittently returns empty list. This has been fixed #5038 #5189

    Other

    • VTTablet: Etcd query timeout because auto-committable transactions are not properly rolled back. #6258 #6299

    • VTTablet: Fixed race in schema engine during notification of schema changes. #6268

    • VTTablet: Reloading ACL config was throwing misleading error messages.his has now been fixed and the error messages improved. #6309

    • VTTablet: Panic during backup because of an error in the pgzip library has been fixed by contributing a fix to pgzip and upgrading to a new release that includes the fix. #6189

    • VTTablet: Improvement of error handling by retrying while uploading to S3. #6182 #6395 #6405

    • VTTablet: builtinbackupengine was uploading a manifest despite errors from S3. Error collection during backup has been fixed to address this. #6349 #6350

    • VTTablet: Fix dba connection leak in LockTables. #6424

    • VTTablet: Fix connection pool leak in schema engine. #6426

    • VTTablet: Fix panic from binlog_streamer when ‘-watch_replication_stream’ is set., #6253

    • Vtctld UI: Change all external links in vitess to use https #6170 #6169

    • Ensure SrvVSchema gets rebuilt for all cells #6276

    • Fix MariadbGTIDSet multi-domain support #6184

    • mysqlctl: Fix connection leak during killConnection() #6245

    • Fix release script to install examples in the correct directory #6159

    Functionality Added or Changed

    VTGate/ MySQL compatibility

    • Transactions on read-only replicas are now supported. Start a transaction by first choosing the desired tablet_type ("use @replica"), then issuing a “BEGIN” statement. Internally, this will issue a “START TRANSACTION READ ONLY” on the underlying database. This is only available if using tabletgateway. #6166 #6244

    • Create better plans for NULL comparisons. Instead of using a SelectScatter, now vtgate will produce empty results when the predicate is a comparison against literal NULLs. #6133 #6425

    • Add EXPLAIN support for vtgate plans. View the produced vtgate query plan by using EXPLAIN FORMAT=VITESS.

    • Added savepoint support. #6355 #6404 #6412 #6414

    • "ignore_nulls" option to lookup vindexes #6147 #6222

    • Add support for row_count() function. #6174 #6179

    • XOR Operator is now supported in sql WHERE clause. #6369 #6371

    • Allow 'binary' for character_set_results. It is equivalent to null. #6237

    • Improve logging by adding keyspace, table, and tablet_type. #6282

    • VTGate now takes in optional flags for ‘max_payload_size’ and ‘warn_payload_size’. A payload size above the warning threshold will only produce a warning. A payload size that exceeds ‘max_payload_size’ will result in an error. The default behavior is not to check payload size, which is backwards-compatible. #6143 #6375

    • A new comment directive (‘/*vt+ IGNORE_MAX_PAYLOAD_SIZE=1 */) has been added to provide the ability to override the payload size checking per query. #6143

    • Support ‘select * from information_schema.processlist’. #6310

    • VTGate: Support batch lookup queries for integer and binary types. Other types will continue to require one query per row. #6420

    • Support system settings at session level. #6149

    • Prepare for future features by adding reserved connections. This is the ability for a connected client to have a dedicated connection instead of using the connection pool. #6303 #6313

    • Allow charset introducers with vindex column values. #6236 #6314

    • Support Union All between different shards. #6351

    • VTGate: Exclude KeyspaceId(s) from plan cache key.This improves the cache hit rate. #6233

    • VTGate: Reload TLS config on SIGHUP. #6215

    • VTGate: Drop vindex DDL. #6151

    • VTGate: Show vitess_tablets now reports master_term_start_time. We ensure that this is reported correctly after reparents. #6135 #6292 #6293

    VReplication

    • VStream Copy extends the VStream API to stream entire databases or tables. Earlier you specified a position to start streaming events from. #6277

    • VReplication workflows now accept "-cells" as an option. This can be a comma-separated list of cells or cellsAliases. The source tablet for vreplication will be chosen randomly among the tablets that belong to the specified cells/cellsAliases. If the option is not provided, we default to the cell that is local to the vreplication target. #6442, #6456

    • Schema Versioning. Tablets now have a schema tracker that can be enabled to store schema versions as DDLs are applied. The _vt.schema_version table contains the schema with the corresponding ddl and gtid position when the ddl was applied. The tracked schemas are used by the new Historian service to provide vstreams with the correct schema while streaming old events. #6300

    • Foreign Keys are now supported on the target database during vreplication. In the copy phase we disable FK checks and re-enable them once the replication phase starts. #6284

    • Materialize: added a drop_constraint option during table copy to strip constraints while creating target tables #6422

    • Materialize: optimizations while creating target tables resulting in a significant improvement in performance when moving a large number of tables from an external MySQL instance #6207, #6251

    • DropSources: Added ‘-rename_tables’ option to rename the source tables rather than dropping them entirely. #6383

    • Add keyspace, shard and workflow labels to VReplicationSecondsBehindMaster metrics #6279

    • Correct command line help #6187

    Point-in-time Recovery

    In the 4.0 release we introduced functionality to recover a keyspace into a snapshot based on a specified time. That feature only allowed you to restore a backup, but did not have the ability to replay binlogs up to the specified time. The feature has now been completed so that true point-in-time recovery is possible. This is currently only supported for MySQL compatible installations, not MariaDB. #4857 #6267 #6408

    Healthcheck

    • VTGate can now use a new healthcheck that is logically simpler while providing the same functionality. This is the default. The older healthcheck has been deprecated but is still available to use if desired by setting "-gateway_implementation=discoverygateway" #5750 #6155

    • VTCombo has been migrated to use the new healthcheck #6302

    MiniVitess

    A docker container that can be used to bring up a self-contained vitess cluster, which can automatically resolve and self-configure to meet an external MySQL cluster. While this might sound very similar to VTCombo and vttestserver, it is not quite the same. MiniVitess runs in docker, brings up distinct processes for each vitess component and works with an external MySQL cluster. #6377

    Examples / Tutorials

    • Vitess operator example #6154

    • Region sharding example #6209 #6275

    • Change local example to use new command DropSources #6178

    • Are-you-alive example now makes it easier to test multiple endpoints #6202

    • Local docker tutorial added to website

    Other

    • Vtctl ListAllTablets now shows the MasterTermStartTime. This is shown as "" for read-only tablet types. In case of an old master that has not yet updated the topo record, TabletType will be reported as “unknown” #6135

    • New debug handlers have been added to set block and mutex profiling. #6137

    • Vitessdriver: Allow overriding driver name. Default stays as "vitess" #6138

    • VTCombo: add options to allow starting MySQL within VTCombo. Note that vttestserver provides similar functionality. #6265

    • Disable foreign key checks during preflight schema change tests #4696

    • Fix labels for opentsdb #6289

    • Allow empty shards when running ValidateSchemaKeyspace behind flag ‘-skip-no-master’ #6216

    • Death by SIGPIPE on stdout/stderr can be avoided by setting the new flag ‘catch-sigpipe’ #6297

    • Field query support in shard targeting #6132

    Documentation

    • Fix links to local example and sample client in vitessdriver godoc #6254

    • Flag documentation has been improved #6261

    • All flags are now documented in the program reference on the website.

    Build Environment Changes

    • golangci-lint now runs as a pre-commit hook and CI check #6385 #6406

    • CI test for region_sharding example #6275

    • Removed staticcheck as a pre-commit hook because it is included in golangci-lint #6416

    • Faster unit tests #6415

    • Consul-api version upgraded to 1.8.0 #6358

    • Upgrade pgzip to v1.2.4 #6189

    • Upgrade log4j2 version to 2.13.3 #6403

    • Add docker_local target to Makefile and a docker/local/run.sh script #6393

    • Build: Only check for k3s on Linux #6353

    Functionality Neutral Changes

    • Add context to MysqlDaemon schema functions. This lets us handle disconnected RPCs instead of the client hanging forever. #6243

    • Refactor table filtering logic #6242 #6259

    • VTGate: Replace Sprintf with string concatenation in normalizer, this reduces CPU usage on benchmarks #6127

    • VTTablet: Reduce the number of packets sent from vttablet to MySQL #6130

    • Fixed flakiness in External Connector Test: fixed race #6201

    • Fixed flakiness in Message Test: test had not been updated after jitter functionality was added #6283

    • Fix flakiness in TestConnectTimeout #6188

    • Fix TLS test for Go 1.13+ #6185

    • Fixed linter errors #6416 #6364 #6307

    • More dollar-sign tests #6363

    • VTExplain: use DiscoveryGateway (and old healthcheck) until migration can be completed #6248

    • Deprecated RPCs from 6.0 have been deleted #6380

    • Update references to orchestrator repo to point to openark instead of github #6360

    • Terminology: deprecate or replace references to "slave" with “replica” except where referring to actual sql commands or fields returned from MySQL. #6428 #6392 #6379

    • Fix log format - use Infof instead of Info #6413

    • Cleanup temp files used in tests #6400

    • Add unit test for func stringMapToString #6280

    • Check errors in etcd2topo unit test #6128

    • Unit test for atomic.go #6120

    • Unit test for ResolveIPv4Addrs #6230

    • Enabled set statement tests #6167

    Source code(tar.gz)
    Source code(zip)
    vitess-7.0.0-beta-e86a967.tar.gz(322.67 MB)
    vitess-7.0.0_beta-e86a967.x86_64.rpm(320.79 MB)
    vitess_7.0.0-beta-e86a967_amd64.deb(322.70 MB)
  • v6.0.20-20200624(Jun 24, 2020)

    This update fixes a regression which was significant enough that it was decided to backport to the release branch.

    Bug fixed

    #6250 added support for float64 durations in vttablet's yaml configuration. The change introduced a regression where the heartbeat interval unit was changed to seconds, but the conversion was not correspondingly changed. In other words, a heartbeat interval of 1s would have resulted in a value of 1000s instead. The net effect is that healthy replicas would be reported as heavily lagged.

    This issue affects only those who enable heartbeats.

    This regression as merged into https://github.com/vitessio/vitess/releases/tag/v6.0.20-20200617. #6359 fixes #6356

    Source code(tar.gz)
    Source code(zip)
    vitess-6.0.20-20200624-9438d77.tar.gz(317.58 MB)
    vitess-6.0.20_20200624-9438d77.x86_64.rpm(315.75 MB)
    vitess_6.0.20-20200624-9438d77_amd64.deb(317.59 MB)
  • v6.0.20-20200617(Jun 17, 2020)

    This update fixes a number of regressions which were significant enough that it was decided to backport to the release branch. The issue with XtraBackup not being compatible with MySQL 8.0.20 has been fixed by Percona. Anyone using MySQL 8.0.20 or higher will need to use Xtrabackup 8.0.12.

    Bugs Fixed

    • Vtgate: Fix panic from select database() in OLAP mode #6318 Fixes #6200
    • Vtgate: Fix DEFAULT function not working with keyword column names #6319 Fixes #6221
    • Vttablet: Fix regression where some options could no longer be specified as floating-point values #6320
    • Vtgate: Fix regression on autocommit transactions that led to extra begin statements being issued, and updates getting rolled back because of a previous error #6321 Fixes #6264 and #6285
    • Security: new versions of Netty and tcnative #6322
    • Vtgate: Fix panic when selecting @variable #6323 Fixes #6294
    • Vtgate: Fix problem with incorrect values being inserted due to bindvariable name collisions #6324 #6327 Fixes #6266
    Source code(tar.gz)
    Source code(zip)
    vitess-6.0.20-20200617-5ba6549.tar.gz(317.60 MB)
    vitess-6.0.20_20200617-5ba6549.x86_64.rpm(315.75 MB)
    vitess_6.0.20-20200617-5ba6549_amd64.deb(317.60 MB)
  • v6.0.20-20200525(May 25, 2020)

    This issue addresses a regression in vtgate performance, which was significant enough that it was decided to backport to the release branch. In addition to this bug fix, there remains a known issue where XtraBackup is not compatible with MySQL 8.0.20. Percona is aware of the issue, and there are no known fixes at this point in time.

    Bugs Fixed

    • Vtgate: a performance regression was identified as part of the refactoring of vtgate's executor. The executor now maintains a cached copy of the vschema. #6220 #6203
    Source code(tar.gz)
    Source code(zip)
    vitess-6.0.20-20200525-5245945.tar.gz(317.58 MB)
    vitess-6.0.20_20200525-5245945.x86_64.rpm(315.72 MB)
    vitess_6.0.20-20200525-5245945_amd64.deb(317.58 MB)
  • v6.0.20-20200519(May 19, 2020)

    This update fixes a number of issues related to backup discovered in Vitess 6, and is a recommended update for all users. In addition to these bug fixes, there remains a known issue where XtraBackup is not compatible with MySQL 8.0.20. Percona is aware of the issue, and there are no known fixes at this point in time.

    Bugs Fixed

    • VTTablet: An issue exists where transaction pool timeout did not apply, and connections never expired #6197
    • VTTablet: An issue was discovered where copying a backup to S3 failed with an error of closed network connection: cannot copy data. This has been worked around by retrying of S3 upload requests. #6199
    • VTTablet: The included version of pgzip has been upgraded to v1.2.4. This helps address an occasional panic that was observed with xtrabackup backups. #6198
    Source code(tar.gz)
    Source code(zip)
    vitess-6.0.20-20200519-f758b52.tar.gz(317.57 MB)
    vitess-6.0.20_20200519-f758b52.x86_64.rpm(315.73 MB)
    vitess_6.0.20-20200519-f758b52_amd64.deb(317.58 MB)
  • v6.0.20-20200508(May 8, 2020)

    Known Issues

    1. While running a builtin backup with large databases, you might sometimes get this error: use of closed network connection: cannot copy data: MultipartUpload: upload multipart failed. We believe this is caused by the newer aws-sdk-go version we upgraded to in 5.0. It might be this problem filed upstream.
    2. On xtrabackup backed up tablets on the new build, you might occasionally hit this panic:
      Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, caught panic during Backup: bufio: writer returned negative count from Write This also seems to be related to aws-sdk-go, though we haven't traced it to a known issue.
    3. There is a known regression where the transaction pool timeout will not apply, and connections never expire. We are working on a fix for this issue, and will release an update shortly.

    Release Notes

    Two regressions were discovered in Vitess 6 after release. While neither are critical, they are visible enough that we wanted to release a patch update:

    • A regression in the packaging of tar.gz files moved the examples from examples/ to share/vitess/examples. This made the install from binary instructions incorrect. #6159 #6160

    • A regression in how select database() returned the current database in use broke how the MySQL command-line client changes the currently selected database. This issue is specific to the mysql client, and is not known to affect application usage of com_init_db or use <dbname> #6157 #6158

    Source code(tar.gz)
    Source code(zip)
    vitess-6.0.20-20200508-147bc5a.tar.gz(317.61 MB)
    vitess-6.0.20_20200508-147bc5a.x86_64.rpm(315.77 MB)
    vitess_6.0.20-20200508-147bc5a_amd64.deb(317.63 MB)
  • v6.0.20-20200429(Apr 29, 2020)

    Known Issues

    1. While running a builtin backup with large databases, you might sometimes get this error: use of closed network connection: cannot copy data: MultipartUpload: upload multipart failed. We believe this is caused by the newer aws-sdk-go version we upgraded to in 5.0. It might be this problem filed upstream.
    2. On xtrabackup backed up tablets on the new build, you might occasionally hit this panic:
      Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, caught panic during Backup: bufio: writer returned negative count from Write. This also seems to be related to aws-sdk-go, though we haven't traced it to a known issue.
    3. There is a known regression where the transaction pool timeout will not apply, and connections never expire. We are working on a fix for this issue, and will release an update shortly.

    Incompatible Changes

    This release includes the following changes, which may result in incompatibilities when upgrading from a previous release:

    Changes to Local Examples

    The local examples have been updated to use the new VReplication workflows for MoveTables and Resharding. This obsoletes the previous workflows of Vertical Split Clone and Horizontal Sharding. The earlier examples remain available in the “local.legacy” example directory for now. #6016 #6015 #6052 #6072 #6106 #6121

    Change to Helm Chart

    The helm chart has been updated to default to using Kubernetes as the Topology Server provider. This change was required to remove the dependency on etcd-operator, which is no longer under active support. #5989

    Earlier Kubernetes Operator Retired

    Support for the Vitess Kubernetes Operator has been removed due to a lack of maintainers. Adoption is understood to be low, but users are encouraged to migrate to using the Helm charts which continue to be supported, or PlanetScale’s Vitess Operator (provided by a third party) #5873

    SQL Support: Keywords are no longer lowercase

    Vitess no longer lower cases SQL keywords. In corner cases, this may affect applications which made specific use of SQL keywords, and then referred to a column in the result set #5897

    SQL Support: Parenthesis are now removed from the AST

    Vitess now simplifies expressions to remove parentheses from expressions. In corner cases, this may affect applications that use an expression that contains parens and then refer to it as a column in the result. #5906

    Statement Based Replication, SplitQuery and UpdateStream

    Code for the unsupported features of Statement-based replication (SBR), SplitQuery and UpdateStream has now been removed. Users using SBR are encouraged to switch to using Row-based replication instead. #5940 #5972 #5928

    Messaging: First Class Citizen + API Improvements

    The messaging feature has had several breaking changes and feature additions

    • Direct inserts are now supported: INSERT INTO msg_table SELECT col1, col2...
    • time_scheduled and time_created are no longer required nor automatically populated.
    • priority is a new required field. Messages with a lower priority will be processed first.
    • Message retention is now measured after time_acked instead of time_scheduled
    • New table options vt_min_backoff and vt_max_backoff set bounds on exponential backoff for message retries
    • 33% backoff jitter has been added for messages that are getting postponed to prevent thundering herds.

    Users of the messages feature are encouraged to read through the full details in issue #5947 #5948 #6114

    RPM and DEB Packages

    Vitess binaries and examples are now installed in system standard paths, rather than the /vt directory #5836

    Vitess Go Driver

    Values of type sqltypes.Timestamp are now automatically converted to time.Time values #5788

    Simplified VTtablet Schema Load Process

    The process of loading vttablet schema has been simplified, as many of the statistics loaded were expensive to gather but remained unused. This may lead to compatibility breakage in the use of the features: Sequences, VStreamer and Messages #5951

    Planned Reparent

    The implementation of Planned Reparent and Emergency Reparent have been improved, with more graceful handling of failure modes. While this change is functionality neutral, it is important that Vitess components are upgraded in the recommended order. #6050

    Bugs Fixed

    • Binlog Client: Vitess components now randomize the server_id when acting as a MySQL replica. #5998

    • heartbeat: Prevent a scenario where tables could be created on a replica leading to errant transactions. Now tables are only created on the master with binlog ON, using the app_user not dba_user #6053

    • Messages: For the mysql protocol, buffered results coming from vitess are flushed a minimum of every 100ms, and configurable by mysql_server_flush_delay. This helps prevent against a case where intermittent streaming could delay messages from being sent. #5879

    • mysqlctl: Correctly escape and quote passwords that contain special characters. #5761

    • VReplication: Several improvements to failure handling and miscellaneous fixes #5753 #5792 #5801 #5842 #5935 #5905 #5845 #5942

    • Vtcompose: Add a check to make sure that files can be read #6024

    • Vtcompose: refactor vtcompose for improved testability #5994

    • Vindexes: Support for a region_json Vindex #6017 #5806 #5955

    • Vtctl: Before applying a VSchema, vtctl now correctly checks that the schema exists. #5931

    • Vtctl: Vtctl now exits early if the topo_global_root was set incorrectly #6119

    • Vtctl: EnsureVSchema and GetOrCreateShard now pass in the cell name. This helps prevent against corruption, which could occur from auto-rebuilding of cells automatically when creating a new shard or keyspace. #5930

    • VTExplain: An issue where incorrectly formatted comments could lead to a hang has been fixed. #5984

    • VTGate: Leading zeros were incorrectly dropped in microsecond timestamps when using prepared statements #5901

    • VTGate: Setting the transaction_mode in a mysql session is now case insensitive. #5963

    • VTGate: When updating consistent indexes, VTGate applies an optimization where it checks if anything really changed before applying an update. In the case that this check results in errors, VTGate will now skip the optimization and execute the update regardless. #5883

    • VTGate: Remove allocation of unused bytes #5874

    • VTGate: The gateway interface functionality has been simplified in anticipation of two new gateway implementations: TabletGateway and VTDirectGateway #5978 #5992

    • VTGate: The process of watching for VSchema updates from the topology server now recovers successfully if it receives an invalid update. This helps prevent against a scenario where a malformed vschema could take down all vtgate servers. #6051

    • VTGate: Previously VTGate performed health checks against all keyspaces, not just keyspaces listed in keyspaces_to_watch. Healthchecks are now limited correctly. #5815

    • VTGate: SQL Support: database() is now only rewritten against dual #5793

    • VTGate: Plan Cache is now disabled for statements with nil instructions #6045

    • VTGate: Make sure to only accept DDL that can be parsed, at least partially #6023

    • VTGate: Log when accepting a new connection fails #5881

    • VTGate: Prepared Statements did not correctly handle signed integers #5974

    • VTGate: Executor statistics did not correctly account for dividing by zero, leading to a potential panic. #5904

    • VTGate: The error message for unsupported DDL creation has been improved #6109

    • VTGate: The metric QueriesProcessedByTable was not correctly incremented with streaming queries processed by vtgate #6057

    • GRPC Client: The default keep alive and keep alive timeout values have been changed from unlimited to 10 seconds. This helps prevent cases where queries could hang while waiting for a failed pod or node. In addition, the defaultMaxMessageSize is now set to 16MB #5922 #3890

    • VTTablet: local_metadata is now populated after an agent has been initialized. This helps prevent against the ambiguous case where both -init_populate_metadata and -restore_from_backup are specified. #5783

    • VTTablet: A hang could occur when shutting down a tablet server when the Topo Server is unavailable. Tablet servers will now time out correctly. #5865

    • VTTablet Health Check: A warning is now logged when the vttablet StreamHealth RPC fails #5917

    • VTTablet Planned Reparent: A new context is now created for calling UndoDemoteMaster. This helps prevent a scenario where the existing context may have expired. #5841

    • VTTablet: A panic could occur when calling NewDBConnNoPool #6086

    • VTTablet: Queries of type STREAM_EXECUTE were inadvertently dropped from logging #6056

    • Zookeeper Topo: Trailing newlines are now ignored when specifying-topo_zk_auth_file #6030

    • Misc: Stats counters would erroneously print warnings to log files when adding a negative value. This warning did not consider the case where counters are reused. #5988

    • Misc: Binary log events with the special name of "mysql.rds_*" are now skipped. This helps improve compatibility with Amazon RDS flavors #5966

    Functionality Added or Changed

    • Docker: sysstat and strace are now bundled in vitess/lite images #5814

    • Docker: new lite images based on Red Hat UBI 7 are now available #6100

    • Example Code: An are you alive example has been added #5646

    • Example Code: The demo application has been migrated from Python to Go #6022

    • Example Code: A set of notes and scripts on how to use Vitess with common frameworks #6034

    • Helm charts: now work with the latest Kubernetes version. An issue where the 308_final.yml script did not delete a tablet has also been fixed. #6048 #6083 #6075 #6074

    • Helm charts: Helm charts now enable proxy_tablets by default #6112

    • Java Client: The deprecated VTGateBlockingConn client has now been removed. The newer VTGateBlockingConnection client should be used instead. #5973

    • Get Started Guides: Examples now use the VReplication workflows MoveTables and Reshard instead of Vertical Split Clone and Horizontal Resharding. #6016 #6015 #6052 #6072 #6106

    • Local Example: Scripts that start services have been refactored to ./scripts/vttablet-up.sh, ./scripts/mysqlctl-up.sh and start only one daemon at a time. The motivation for doing this was to make the usage more closely mimic init scripts. #5808

    • Messaging: add new priority field #5967

    • Messaging: Setting both a minimum and maximum backoff time is now supported (rather than exponentially backing off forever) #5979

    • Messaging: Messaging has been refactored to use Row-based Replication vstreamer feed #5913

    • Messaging: A 33% Jitter has been added to postpone backoff #6092

    • Topology Service: A new Kubernetes native Topology Server is now available. #5703 #5957

    • vitessdriver: add ability to set custom gRPC dial options #5820

    • VReplication: Several Improvements to the stability of Vstreamer stability and minor bug fixed. #5878 #5926 #5891 #5804

    • VReplication: A debugging interface is now provided at /debug/vrlog #5889

    • VReplication: The workflows previously known as 'Migrate', 'MigrateReads' and 'MigrateWrites' have been renamed to MoveTables, SwitchReads and SwitchWrites respectively. This breaking change was made because VReplication was previously considered experimental. The new name disambiguates the use of 'Migrate'. These commands also now support a ‘Dry run’ option. #6018

    • VReplication: Filtered VReplication now supports int and varbinary filters for columns #5949

    • VStream: The Vstream now provides journal events, using a similar interface as VReplication. #5746

    • Vtctl: A -force flag has been added so that calls to DeleteCellInfo can still succeed after a topology server has been permanently shut down. #6013

    • Vtctl: A --skip-verify option has been added to CopySchemaShard. This helps in scenarios where source and destination use different versions of MySQL, and verifying the schema as identical may fail. #5265

    • Vtctld: The flag -proxy_tablets has been added. This allows vtctld act as a reverse proxy for all of the tablets it manages. This helps ease a pain point for Kubernetes users #6058 #6062

    • Vtctld: A new health check (/debug/status) has been added to check the status of tablet servers #6113

    • Vtctld: Specifying the cell when initializing vtctld is now optional. If you don't specify a cell, vtctld will choose one for you. #6111

    • VTGate: Improvements to SQL Support:

      • For compatibility with mysqldump, the syntax SET unique_checks=0 and SET sql_quote_show_create=1 is now passed but ignored #5846 #5856
      • The syntax SET SESSION transaction_read_only = 0 is now supported #5827
      • USE INDEX (without index name) #5811 #5763
      • SHOW INDEXES ..[EXTENDED] and SHOW KEYS #6006 #5936
      • SHOW CREATE TABLE with qualifier #5585
      • SHOW FULL COLUMNS #6078
      • SHOW CHARACTER SET WHERE #5710
      • GROUP_CONCAT() with a LIMIT is now permitted #5871
      • SELECT options can now appear in any order #6082
      • Correctly handle DML without a WHERE clause #6011
      • Support for DML scatter queries with lookup vindexes #5829
      • Function names are now formatted correctly #5918
      • Ensure correctness when using SCHEMA() and FOUND_ROWS() functions #5732
      • Ensure correctness of found rows #5924 #5923
      • Improved testing for the use of last_insert_id() and rollback #5887
      • Support for user defined variables #5921
      • Support for partial update of a composite owned Vindex #5875
      • Improve Update and Delete Query Routing for IN Clause #6116
      • Allow SET user defined variables #6093
      • The parser and executor have been incrementally refactored for both maintainability and generating query plans #5770 #5892 #5941 #5995 #6031 #6010 #6090 #5995 #5941 #6035 #6069 #6042 #6047 #6080 #6055 #6064 #6099 #6054 #6108 #6105 #5982 #5959 #5993 #6077 #6096
    • VTGate: When a batched insert fails, VTGate will now only return a list of problem rows that failed due to vindex.Verify errors. Rows that did not produce problems are now omitted. #5896

    • VTGate: The ability to set a default workload is now possible, via -mysql_default_workload #5888

    • VTGate: The previously deprecated v2 routing API has been removed. However, v2 routing is still supported in the limited context of explicitly targeting shards. In the context of explicitly targeting shards, autocommit is now supported and routing has become more flexible. #5962 #5909 #6040

    • VTGate: Client connections are now permitted to upgrade their transaction_mode (for example from MULTI to TWOPC) on a per-session basis #6049

    • VTTablet: The Vttablet Server has been refactored and componentized. This change is a precursor to upcoming features such as multiple schemas per tablet #6027 #6020 #6009 #5809 #6094 #6070

    • VTTablet: A new option named db_connect_timeout_ms has been added to specify the amount of time to wait when connecting to a MySQL Server over TCP. For backwards compatibility, the default is to hang waiting indefinitely. #5832

    • VTTablet Backup: Adding Azure Blob backup support #5674 #5937

    • VTTablet Backup: The option -xtrabackup_prepare_flags has been added to support passing flags to xtrabackup for the prepare phase #5954

    • VTTablet Backup: Previously deprecated flags gcs_backup_storage_project and lock_timeout have been removed #5748

    • VTTablet: A new option -enable-consolidator-replicas has been added to the query engine #5862

    • VTTablet: Error codes returned by the Tablet Server have been improved #6012

    • VTTablet: The option table-acl-config-reload-interval has been added to reload the file specified by -table-acl-config on a timer. This matches similar functionality already available in vtgate. #5925

    • VTTablet: Logging messages have been added during the init tablet phase #6004

    • VTTablet: Resource pools now provide more detailed statistics of wait time distribution #5727

    • VTTablet: Logging has been improved during init tasks. This helps improve debugging failed operations. #6004

    • Misc: The DBConfigs code has been refactored. This helps ensure proper usage of connection parameters for every mysql connection, and cleans up a previous source of bugs. #5833 #5819

    • Misc: All binaries distributed by Vitess now print their version to the log files on startup #5996 #6005

    • Misc: Timeouts are now set for all connection pools #6063

    Build Environment Changes

    • The minimum version of go required to build Vitess has been bumped to 1.13 #5943

    • The consul dependency version has been bumped to 1.5.1 to address CVE-2019-12291 and CVE-2018-19653 #5831

    • The Java Netty dependency has been updated #5848

    • All remaining Python tests have been converted to Go, and Python has been removed as a dependency #5953 #5983 #5999 #5908 #6025 #5805 #5784 #5774 #5790 #5919 #5838 #5826 #5799 #5795 #5866 #5885 #5798 #5794 #5834 #5837 #5884 #5956 #5855 #5958 #5961 #5934 #5916

    • A fix to the Makefile in the previous release of Vitess inadvertently changed the 'make test' target to only run unit tests. This has been reverted. To run the unit tests, execute 'make unit_test' #5786

    • The k3s binary is now installed by bootstrap.sh on Linux platforms #5987

    • Fix unit tests for MySQL 8.0.19 #5803

    • Reduce Flakyness of Unit and End to End tests #6001 #6033 #5880 #6043 #6061 #6076 #6101 #6038 #6091

    • The unused scripts in tools/sauce_connect_*.sh have been removed #5602

    • Code coverage analysis on tests is now available #5849 #5702

    • Fix spelling errors, tests, or other function neutral changes #5986 #5886 #5824 #5828 #5857 #5797 #5933 #5939 #5944 #6095 #6073

    • The pre-commit hook for golangci-lint has been disabled #5952

    • The queries used by Wordpress have been added as an endtoend test #6081 #5898

    • The unused package mysqlproxy has been removed from sources #6079

    Source code(tar.gz)
    Source code(zip)
    vitess-6.0.20-20200429-f7fa695.tar.gz(317.65 MB)
    vitess-6.0.20_20200429-f7fa695.x86_64.rpm(315.80 MB)
    vitess_6.0.20-20200429-f7fa695_amd64.deb(317.65 MB)
  • v5.20+20200204(Feb 4, 2020)

    Incompatible Changes

    This release includes the following changes, which may result in incompatibilities when upgrading from a previous release:

    Deprecated Support for Vagrant Support for deployment on Vagrant is deprecated, and will be removed in the next release of Vitess. The recommended upgrade path is local deployment on macOS.

    Deprecation of vtgate option disable_local_gateway The option disable_local_gateway is parsed but ignored by vtgate. It is now deprecated, and will be removed in a future version #5741

    Web_dir2 and web_dir options for vtctld The options web_dir2 and web_dir are now parsed but ignored. This is a behavior change from 5.0 beta, where the options were removed. We intend to remove these options again in Vitess 6.0, but discovered compatibility issues with existing tooling. #5744 #5747

    Bugs Fixed

    • Consul: The bundled consul version has been bumped to 1.5.1, addressing CVE-2019-12291 and CVE-2018-19653.
    • VReplication: Fixed a panic when connecting to an external MySQL server #5753
    • Vttablet: Address #5734 by forming MySQL error packet on query parse error during
    • Vtgate: Fixed an issue with prepared statements returning a syntax error #5735
    • Mysqlctl: Fixed a regression where the option mysqlctl_mycnf_template was accidentally removed during refactoring. #5717
    • Mysqlctl: Fix a regression where refactoring configuration files removed the options relay-log and relay-log-index from included configuration #5745
    • Vttablet: Fixed a problem with vttablet consuming too much memory during a restore from backup when running with option -s3_backup_log_level LogDebugWithRequestRetries #5777

    Functionality Added or Changed

    • Vtctld: The reference to web/vtctld has been removed from the Makefile #5778
    • mysqlctld: An option has been added to mysqlctl to allow passdown of LD_PRELOAD libraries. This is useful in the case where a custom memory allocation library needs to be specified. #5730
    • Logging: Add datadog plugin for tracing #5681
    • Add support for DISTINCTROW as an alias for DISTINCT SQL syntax #5726 #5718
    • Vstream: support default values #5736
    • VReplication: Add support for ExternalizeVindex #5716
    • ApplySchema: Add support for ALTER DATABASE #5733

    Build Environment Changes

    • Function neutral improvements to code #5766 #5719 #5690
    • Improvements to tests #5759 #5754 #5737
    • A number of tests have been migrated from Python to Go. These changes are function neutral #5743 #5738 #5722 #5684 #5670 #5693
    Source code(tar.gz)
    Source code(zip)
    vitess-5.20+20200204-17a806ae5.tar.gz(247.91 MB)
    vitess-5.20+20200204-17a806ae5.x86_64.rpm(246.52 MB)
    vitess_5.20+20200204-17a806ae5_amd64.deb(247.92 MB)
  • v5.0.1(Jan 15, 2020)

    Incompatible Changes

    This release fixes an issue discovered after releasing Vitess 5.0.0 Beta (#5711). We recommend reading the release notes for 5.0.0 when upgrading from an earlier release.

    Bugs Fixed

    • mysqlctl: Vitess programs would not run if configuration files could not be located #5711
    • VReplication: The keyrange was incorrectly being quoted twice #5715
    • Release packaging: Fix issues discovered in automated release scripts #5714
    • Docker images: provide a workaround for missing configuration files in vitess/lite image #5707
    • Change http links to https #5697
    • helm: Fix examples to remove undefined flag #5696

    Functionality Added or Changed

    • VReplication: CreateLookupVindex command with backfill #5692
    Source code(tar.gz)
    Source code(zip)
    vitess-5.0.1-5165f85.tar.gz(239.50 MB)
    vitess-5.0.1-5165f85.x86_64.rpm(238.16 MB)
    vitess_5.0.1-5165f85_amd64.deb(239.50 MB)
  • v5.0.0(Jan 13, 2020)

    Known Issues

    Important! This release contains a bug, where MySQL configuration files were intended to be bundled in the mysqlctl binary, are not correctly bundled. The workaround is to ensure that $VTROOT/config is available on your system. We intend to fix this issue before final release of Vitess 5.0.

    Incompatible Changes

    This release includes the following changes, which may result in incompatibilities when upgrading from a previous release:

    Removal of binaries from release packages The release packages have been improved to no longer ship unneeded dependencies. If you previously installed etcd, zookeeper or consul as part of the Vitess package, you will now need to download and install it separately. #5545 #5645

    Refactoring MySQL Configuration Files The included my.cnf files for MySQL have been simplified and improved. There may be some isolated cases where the difference in configuration may lead to compatibility issues with a previous installation of MySQL. #5326 #5517 #5704

    VTROOT, VT_MYSQL_ROOT, VTDATAROOT changes You are no longer required to specify the environment variables VTROOT, VT_MYSQL_ROOT, VTDATAROOT in order to run the Vitess examples. These variables have been changed to autodetect. Use of these variables is still permitted, but discouraged, as they may be removed in a future release. #5488 #5506 #5654 #5560

    Elimination of VTTOP environment variable The environment variable VTTOP has been eliminated, with all uses changed to VTROOT instead. In most cases VTROOT can also be autodetected, so it is encouraged to remove VTTOP from any scripts. #5527 #5632 #5546 #5548

    Removal of vtqueryserver The unused vtqueryserver utility has been removed. #5583

    web_dir2 and web_dir options removed from vtctld The web files used by vtctld are now embedded into the vtctld binary, simplifying configuration. This results in a behavior change since web_dir and web_dir2 are no longer valid parameters when starting vtctld. Please be sure to update your startup scripts accordingly. #5597

    Examples now check for dependencies The examples now check for shell programs that are required to execute correctly. If you do not have the following programs in your PATH, it will result in an error: mysqld etcd etcdctl curl vtctlclient vttablet vtgate vtctld mysqlctl #5543

    Changes to build from source The build from source instructions have changed. See https://vitess.io/docs/contributing/build-from-source/ for details.

    Bugs Fixed

    • The stability of prepared statements has been improved #5562 #5441
    • The portability of shell scripts has been improved. #5384 #5480
    • The semi-sync extension is now optional #5483
    • A panic could occur when a sequence name was bad. #5621
    • Loading TLS and x509 certificates now consumes less memory. #5423
    • Helm no longer complains about nil values in templates. #5484
    • Add workaround for common MySQL replication error #5627
    • Add null as a valid constant SELECT value for vtexplain #5624
    • Creating new shards could overwhelm a topo server. Vitess now no longer performs actions concurrently when creating shards in the topo. #5436
    • VTExplain could race when waiting around for the fakesqldb tabletserver #5476
    • VTGate now correctly resolves columns in information_schema tables. #5608
    • The query SHOW FULL COLUMNS now works correctly with a filter #5558
    • Backups now pass the correct filesize to Ceph #5369
    • Vtbackup will now stop replication after reaching the goal position. #5437
    • The bundled version of log4j has been upgraded #5691
    • VTTablet now accepts external tablets in Azure MySQL #5683
    • VTGate now advertises protocol support for FOUND ROWS #5676
    • A buffered reader is now placed in front of xtrabackup’s STDOUT pipe, potentially reducing memory usage #5666
    • Consultopo: Enforce a connection heartbeat for Watch() #5673
    • The tabletmanager now ensures that it is fully shutdown before closing the topology client #5667

    Functionality Added or Changed

    • Several improvements to VReplication #5432 #5628 #5367 #5594 #5575 #5574 #5509 #5500 #5498 #5494 #5491 #5457 #5456 #5454 #5429 #5490 #5515 #5689 #5688 #5669 Including:
      • Support for file+pos based replication, allowing replication in from MySQL instances which do not have GTIDs enabled
      • New Materialize command
      • Support for Vdiff, Reshard
      • Vindex and multi-column support
    • Several Improvements to Docker files. #5592 #5510 #5470 #5458 #5444 #5492 #5661 Including:
      • A refactoring of vitess/lite files
      • Support for aarch64
      • The default mysql data files are now removed from the vitess/lite images
      • The latest xtrabackup is always retrieved
      • Jemalloc and tcmalloc are now bundled.
    • Mysqlctl now checks that it is not being executed as root. This helps debugging scenarios, since mysqld will refuse to execute as root by default. #5419 #5644
    • The examples have been improved. Improvements include a check that all dependencies are installed, and warn if it looks like selinux or apparmor prevented mysqlctl from starting #5543 #5573 #5461 #5433 #5426 #5386
    • The bootstrap process will search for etcd in the PATH, and won’t install it if it is present. In addition, etcd on aarch64 is now supported. #5427 #5481 #5428 #5495 #5434 #5425
    • VTGate now supports proxy protocol #5563
    • There is now an option for TLS enabled etcd server connections. #5496
    • Vtgate now advertises itself as a MySQL 5.7 server #5482
    • A vt_monitoring user has been added to init_db.sql. #5486
    • Mysqlctl now supports bootstrapping MariaDB 10.4. Please note that MariaDB 10.4 is not yet supported. #5374
    • VTGate now provides a scatter of query statistics #5422
    • Vttablet now provides better granularity for errors that come out of the transaction pool #5373
    • Experimental support for region-aware vindexes, and xxhash vindexes #5390 #5439
    • VTTablet servers now support an option to disable caching field names to avoid schema mismatches #5572
    • VTGate: Query routing has been improved when using LEFT JOIN queries #5551
    • VTGate: The parser now supports additional types of BEGIN statements #5550
    • VTTablet servers no longer rechecks the mysql port if it was explicitly provided. This helps ensure operation in proxied environments. #5383
    • The 64 bit hash implementation has been changed from 3DES to DES #5529
    • Demote a master vttablet to a SPARE instead of REPLICA is now supported #5520
    • Vttestserver now supports vschema operations via vtgate #5582
    • The vtgate resolver has been simplified #5679

    Build Environment Changes

    • Several CI tests have been migrated to GitHub Actions, and unit tests are now run against all supported MySQL flavors and versions. #5651 #5635 #5559 #5561 #5414 #5557 #5487
    • Several endtoend tests have been converted from Python 2.7 to Go, and now run via GitHub Actions. #5650 #5643 #5630 #5619 #5607 #5604 #5596 #5595 #5587 #5566 #5513 #5479 #5694 #5664
    • Improvements to reduce the Flakiness of tests #5642 #5639 #5637 #5625 #5622 #5620 #5615 #5612 #5567 #5648 #5633 #5658 #5657
    • The unit race test now tests all go source files #5629 #5499
    • Releases are now automated via GitHub Actions #5593 #5589
    • The unused files in tools/sauce_connect_*.sh have been removed #5602
    • The testsuite now defaults to using Row-based Replication #5588
    • Improvements to Git hooks #5544 #5453
    • Out of date references to keytar and broken links have been fixed. #5536
    • The build system will no longer warn when go is installed via a system package. #5407
    • The build system now explicitly sets GOBIN to VTROOT/bin #5468
    • Minor code improvements #5501 #5552 #5435 #5526 #5601 #5541 #5472 #5404 #5678 #5663
    • Update vagrant bootstrap to install pip #5440
    Source code(tar.gz)
    Source code(zip)
    sha256_manifest.txt(345 bytes)
    vitess-5.0.0-1b384b8a7.tar.gz(239.39 MB)
    vitess-5.0.0-1b384b8a7.x86_64.rpm(238.05 MB)
    vitess_5.0.0-1b384b8a7_amd64.deb(239.40 MB)
simply SQL Parser for Go ( powered by vitess and TiDB )

vitess-sqlparser Simply SQL and DDL parser for Go (powered by vitess and TiDB ) this library inspired by https://github.com/xwb1989/sqlparser (origina

BlasTrain Co., Ltd. 244 Nov 28, 2021
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置,参数n为秒数,查询结

null 11 Nov 1, 2021
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

PingCAP 247 Nov 29, 2021
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice.

DoltHub 662 Dec 5, 2021
Interactive terminal user interface and CLI for database connections. MySQL, PostgreSQL. More to come.

?? dbui dbui is the terminal user interface and CLI for database connections. It provides features like, Connect to multiple data sources and instance

Kanan Rahimov 64 Nov 30, 2021
Golang restAPI crud project with mySql database.

Golang RestAPI using gorilla/mux Golang restAPI crud project with mySql database. Test Api with Thunder Client vs code beautiful Extension. and use Be

Md Abu. Raihan 5 Oct 9, 2021
A Go rest API project that is following solid and common principles and is connected to local MySQL database.

This is an intermediate-level go project that running with a project structure optimized RESTful API service in Go. API's of that project is designed based on solid and common principles and connected to the local MySQL database.

Kıvanç Aydoğmuş 12 Sep 23, 2021
CRUD API example is written in Go using net/http package and MySQL database.

GoCrudBook CRUD API example is written in Go using net/http package and MySQL database. Requirements Go MySQL Code Editor Project Structure GoCrudBook

Serhat Karabulut 1 Dec 3, 2021
[mirror] the database client and tools for the Go vulnerability database

The Go Vulnerability Database golang.org/x/vulndb This repository is a prototype of the Go Vulnerability Database. Read the Draft Design. Neither the

Go 15 Nov 22, 2021
a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

siddontang 3.3k Nov 27, 2021
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.6k Dec 6, 2021
A high-performance MySQL proxy

kingshard 中文主页 Overview kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read

Fei Chen 5.9k Dec 3, 2021
Golang MySql binary log replication listener

Go MySql binary log replication listener Pure Go Implementation of MySQL replication protocol. This allow you to receive event like insert, update, de

Pavel <Ven> Gulbin 176 Oct 11, 2021
MySQL replication topology management and HA

orchestrator [Documentation] orchestrator is a MySQL high availability and replication management tool, runs as a service and provides command line ac

null 4.3k Nov 29, 2021
db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

null 19 Nov 5, 2021
一个使 mysql,pgsql 数据库表自动生成 go struct 的工具

db2go 一个使 mysql、pgsql 数据库表自动生成 go struct 的工具 快速使用 将项目放入到GOPATH/src目录下

易水韩 17 Sep 28, 2021
🐳 A most popular sql audit platform for mysql

?? A most popular sql audit platform for mysql

Henry Yee 5.9k Nov 26, 2021
GitHub's Online Schema Migrations for MySQL

gh-ost GitHub's online schema migration for MySQL gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides paus

GitHub 9.5k Dec 2, 2021