open source training courses about distributed database and distributed systemes

Overview

Welcome to learn Talent Plan Courses!

Talent Plan Logo

Talent Plan is an open source training program initiated by PingCAP. It aims to create or combine some open source learning materials for people interested in open source, distributed systems, Rust, Golang, and other infrastructure knowledges. As such, a series of courses focused on open source collaboration, rust programming, distributed database and systems are provided.

Note:

Each course is developed independently, so they vary in their presentation and their expectations from course-takers. Please see the individual course documentation for details.

Our Courses

Series 1: Open Source Collaboration

Open source collaboration includes a series of open source related learning mateials to help open source enthusiasts have a basic knowledge of what open source software is, the differences among kinds of open source software licenses, how to participate in open source projects and what a welcoming open source community looks like. Courses of this series are:

Series 2: Rust Programming

Two courses are included in this series, which are:

  • TP 201: Practical Networked Applications in Rust. A series of projects that incrementally develop a single Rust project from the ground up into a high-performance, networked, parallel and asynchronous key/value store. Along the way various real-world and practical Rust development subject matter are explored and discussed.

  • TP 202: Distributed Systems in Rust. Adapted from the MIT 6.824 distributed systems coursework, this course focuses on implementing important distributed algorithms, including the Raft consensus algorithm, and the Percolator distributed transaction protocol.

Series 3: Distributed Database

Two courses are included in this series, which are:

Series 4: Deep Dive into TiDB Ecosystems

  • TP 401: Deep Dive into TiDB(WIP)
  • TP 402: Deep Dive into TiKV(WIP)

See Courses for more details

Contributing to talent plan

Contributions of any kind are welcome! Check out the Contributing Guide in this repository for more information on how you can contribute to talent plan.

We love our community and take great care to ensure it is fun, safe and rewarding. Please review our Code of Conduct for community expectations and guidelines for reporting concerns.

We're here to help

If you have questions about building (or taking) courses, you can ask in the channel #wg-talent-plan-courses of the tidbcommunity slack workspace.

License

These courses may be freely used and modified for any purpose, under the terms of each course's individual license. See the courses for details.

Issues
  • Create initial readme for Rust course and outline lesson plan

    Create initial readme for Rust course and outline lesson plan

    This PR describes a Rust training course called "Practical Networked Applications in Rust", and lays out a draft lesson plan.

    Rendered README.

    Rendered lesson plan.

    The flow of this course is curious, with topics derived from the needs of the hands-on projects and the hands-on projects designed to teach useful and interesting topics, leaving some important topics not covered (at least yet). I believe the lesson plan proceeds quite differently from typical Rust intro/beginner material. I hope that the documentation conveys the rationale sufficiently.

    It is not quite finished, but I think there's enough here to understand what it's about, enough to review and critique.

    The structure of this course is inspired by the MIT 5.824 distributed systems course, which we are also adapting to Rust as a follow-on course to this one. I intend that the two courses ultimately share a common structure.

    The lesson plan is intended to be an index, with links to the full lessons and projects. The descriptions within the lesson plan of the goals, topics etc are intended as a guide to writing the full lessons and project descriptions. Once those are written we might reduce the descriptive content in the lesson plan so that it's easier for the reader to take in the full lesson/project progression by skimming.

    I very much want feedback. Are the goals and non-goals reasonable? Is the project interesting? Does the lesson plan make sense? Are the topics interesting? Within the given lesson plan, and considering the projects being implemented, what other topics should be included? As a new Rust programmer, does the content here excite you to take the course? What changes would make the course more exciting?

    I'm also looking for people interested in helping build this course, since there's quite a lot of material to cover. If this looks like something you would like to participate in, please speak up.

    Note that there is a file here called notes.md that contains notes I've taken about this course. It includes a list of potential subjects and links to other courses and draft training material. I intend the material to take inspiration, if not content, from https://github.com/ferrous-systems/rust-three-days-course, https://github.com/nrc/talks, and other existing training material. We'll do a survey of existing training material soon, with goals of understanding how this course can provide unique value, and how other courses progress from topic to topic.

    I was hoping that this PR would fill out some of the material's structure, like how the slides and text are organized, an example full project description, how the projects' tests work; but I haven't made it that far yet.

    cc @siddontang PTAL. I hope that the projects are aligned with your expectations. I suspect the volume of lesson material here is more ambitious than you expected, and is very probably more than can be completed in Q1. We can deprioritize certain lessons for an MVP. Give me all the feedback you've got. We haven't discussed how this course is licensed or who owns it, but I've preemptively put a CC-BY notice at the bottom of the readme so that there's some intent conveyed as to the license.

    cc @Hoverbear @breeswish @overvenus @rleungx @Hijiao @huachaohuang @nrc if you have the time I'd be grateful for your feedback.

    cc @skade I'd be grateful for any review you can offer. Hoverbear and I have been inspired by your rust-three-days-course and you've got lots of experience in Rust training.

    opened by brson 13
  • ./rust code update and ./dss/raft description update

    ./rust code update and ./dss/raft description update

    • Practical Networked Applications in Rust
      • Update: BufReaderWithPos ~~and BufWriterWithPos~~.
      • Fix: Temp_dir is dropped during the test.
    • Distributed Systems in Rust
      • Update: README.md
      • Fix: wrong code in the comment.
    opened by HunDunDM 7
  • rust/docs: correct link for rust-three-day-course

    rust/docs: correct link for rust-three-day-course

    Signed-off-by: Muhammad Falak R Wani [email protected]

    component/rust status/LGT1 contribution status/require-change 
    opened by mfrw 7
  • More rust training content

    More rust training content

    It's been a long time since I've submitted anything, so here's my current branch.

    The main things to note here is that project 1 has more text now, and that I've removed gRPC and tokio from the plan (from discussion with Liu we agreed they were too ambitious for these 5 projects and could come in a later project).

    Rendered

    Tomorrow I will create an issue outlining the next steps so that others might help.

    PTAL @overvenus @nolouch

    opened by brson 6
  • Add tests and an example solution to Rust training Project 1

    Add tests and an example solution to Rust training Project 1

    This pr is based on #7 .

    The structopt extension is not included yet.

    I don't know how to test the cli in integration tests, so I only write tests for KvStore.

    The clap get_matches function uses os_args, which I cannot easily hack.

    Another way is to use get_matches_from_safe and pass args in, but that will make the signature of our main function horrible to beginners.

    assert_cli or assert_cmd should work too. But then I cannot think of the advantage of placing the user interface into the library.

    opened by sticnarf 6
  • Outline Rust project 4

    Outline Rust project 4

    Fixes https://github.com/pingcap/talent-plan/issues/38

    I put this together quickly after our discussion @sticnarf. I do have a few concerns:

    • there is apparently no production-quality concurrent map type we can incorporate :-/
    • like in the previous project, this abstracts a thing then compares to implementations. I don't know if it is valuable to do the same type of exercise again. I'd prefer if we can find a new angle on this, so I've suggested we do a compile-time abstraction using cargo features, but I'm open to any ideas.
    opened by brson 5
  • Project 2 description

    Project 2 description

    Fixes https://github.com/pingcap/talent-plan/issues/11

    opened by brson 5
  • rust/docs: change `sytems` to `systems` to fix a typo

    rust/docs: change `sytems` to `systems` to fix a typo

    Corrects the link to the github "rust-three-days-course" project.

    component/rust status/LGT1 contribution 
    opened by fser 5
  • rust/projects: specify tests

    rust/projects: specify tests

    To accomplish #118

    Specify which tests should be pass in each parts.

    Before modifying the code in subsequent projects, I want to hear from @sticnarf @brson . If you think it's ok, I will add commits to change the code in Project these project.

    I'm not clear in the problems below:

    • In project2, we need to implement set/rm in part2, but some tests for set/rm use get, which should be implemented in part3.
    • In project3, I only specify the tests in part4 and part6. I wonder if there are some more proper way to place these tests.
    component/rust contribution status/require-change 
    opened by mapleFU 5
  • Tidb week 1 homework: single thread merge sort already faster than sort.Slice

    Tidb week 1 homework: single thread merge sort already faster than sort.Slice

    I found in my computer, sort.Slice is about 15s, merge sort in single thread need about 9s. Then I write a base quick sort in hand with go, it is only about 7s in benchmark(sort.Slice slower than hand quick sort maybe because reflect?). Anybody see the same phenomenon?

    Implementions is as follow(go version: 1.12.2):

    merge sort:

    // MergeSort performs the merge sort algorithm.
    // Please supplement this function to accomplish the home work.
    func MergeSort(src []int64) {
    	dst := make([]int64, len(src))
    	copy(dst, src)
    	mSort(dst, src, 0, len(src))
    }
    
    //从src归并到dest
    func mSort(src []int64, dest []int64, low int, high int) {
    	if high-low <= 1 {
    		return
    	}
    
    	mid := (low + high) >> 1
    	mSort(dest, src, low, mid)
    	mSort(dest, src, mid, high)
    
    	for i, p, q := low, low, mid; i < high; i++ {
    		if q >= high || (p < mid && src[p] < src[q]) {
    			dest[i] = src[p]
    			p++
    		} else {
    			dest[i] = src[q]
    			q++
    		}
    	}
    }
    

    my hand quick sort:

    func quickSort(arr []int64) {
    	qSort(arr, 0, len(arr) - 1)
    }
    
    func qSort(arr []int64, low int, high int) {
    	if low < high{
    		pivot := partition(arr, low, high)
    		qSort(arr, low, pivot - 1)
    		qSort(arr, pivot + 1, high)
    	}
    }
    
    func partition(arr []int64, low int, high int) int {
    	pivotKey := arr[low]
    	for low < high{
    		for low < high && arr[high] >= pivotKey{
    			high--
    		}
    		arr[low] = arr[high]
    		for low < high && arr[low] <= pivotKey{
    			low++
    		}
    		arr[high] = arr[low]
    	}
    	arr[low] = pivotKey
    
    	return low
    }
    

    benchmark:

    func BenchmarkMergeSort(b *testing.B) {
    	numElements := 16 << 20
    	src := make([]int64, numElements)
    	original := make([]int64, numElements)
    	prepare(original)
    
    	b.ResetTimer()
    	for i := 0; i < b.N; i++ {
    		b.StopTimer()
    		copy(src, original)
    		b.StartTimer()
    		MergeSort(src)
    	}
    }
    
    
    func BenchmarkNormalSort(b *testing.B) {
    	// 16M个整数
    	numElements := 16 << 20
    	src := make([]int64, numElements)
    	original := make([]int64, numElements)
    	prepare(original)
    
    	b.ResetTimer()
    	for i := 0; i < b.N; i++ {
    		b.StopTimer()
    		copy(src, original)
    		b.StartTimer()
    		sort.Slice(src, func(i, j int) bool { return src[i] < src[j] })
    	}
    }
    
    func BenchmarkQuickSort(b *testing.B) {
    	// 16M个整数
    	numElements := 16 << 20
    	src := make([]int64, numElements)
    	original := make([]int64, numElements)
    	prepare(original)
    
    	b.ResetTimer()
    	for i := 0; i < b.N; i++ {
    		b.StopTimer()
    		copy(src, original)
    		b.StartTimer()
    		quickSort(src)
    	}
    }
    

    Benchmark results as follows:

    goos: linux
    goarch: amd64
    pkg: pingcap/talentplan/tidb/mergesort
    BenchmarkMergeSort-12                 1         9503371329 ns/op
    BenchmarkNormalSort-12                1       15261800118 ns/op
    BenchmarkQuickSort-12                  1         7552893791 ns/op
    PASS
    ok      pingcap/talentplan/tidb/mergesort       37.860s
    

    We can see that merge sort is faster than sort.Slice,hand quick sort is faster than merge sort.

    I also benchmark sort.Sort, found it is slower than sort.Slice

    If so, Is it more meaningful to compare parallel merge sort to quick sort in hand, instead of sort.Slice?

    opened by DQinYuan 5
  • Fixed

    Fixed "prost" compilation issues

    Bump prost version from 0.6 to 0.8.

    Compilation issues in the old version.

    Furthermore some Clippy warnings where fixed.

    What problem does this PR solve?

    • Issue number: close #xxx
    • Breif description of the problem:

    What is changed and how it works?

    Check List

    Tests

    • Unit test
    • Integration test
    • Manual test (add detailed scripts or steps below)
    • No code

    Side effects

    • Possible performance regression
    • Increased code complexity

    Related changes

    • Need to update the documentation
    opened by davxy 0
  • TinySQL suggestions on document writing

    TinySQL suggestions on document writing

    1. write this in the home page of tinySQL: You are advised to use Linux instead of Windows
    2. Project1: explains in more detail on how to test: Clone the project, go to the project directory and run Go Test xxxxx
    3. Project2: write this prompt in the .md file of project 2: After each modification of the . y file, first make parser and then runs the test function
    4. Project3: The brief introduction of the schema change paper is too brief. There is a link on Zhihu to interpret the paper which you can recommend reading it. Example like: https://zhuanlan.zhihu.com/p/309831009. It is neither too brief nor too tedious.
    5. project4: TinySQL's Prerequisit doesn't say you need to learn the optimizer ahead of time. But Project4 went straight to the introduction of the optimizer in TiDB, which was confusing. Please tell beginners what to learn to pass it.
    type/enhancement 
    opened by DylanYang00 0
  • dss: add Raft Lab 2D code

    dss: add Raft Lab 2D code

    What problem does this PR solve?

    • Breif description of the problem: Add lab 2D code from MIT 6.824 (Spring 2021).

    What is changed and how it works?

    • Add 1 test for lab 2A.
    • Add 5 tests for lab 2D.
    • Add 2 interfaces between service and Raft: snapshot and cond_install_snapshot.
    • Update Config for handling snapshot apply messages.

    Here is my solution with these changes that can pass lab 2D: https://github.com/wangrunji0408/talent-plan/tree/0362b3abcd6731649284b32b408a743273a1b63f

    Check List

    Tests

    • Unit test
    • Integration test

    Side effects

    • None

    Related changes

    • Need to update the documentation
    opened by wangrunji0408 0
  • TP 201: Project 3 CLI Requirement

    TP 201: Project 3 CLI Requirement

    Feature Request

    In project 3 for TP 201, the specs say that the --addr field should show up last for kvs-client. I think this isn't the best design because it unnecessarily complicates the subcommands set get and rm by forcing us to append the --addr field to each of them. (see the reference solution). Instead, if placed at the beginning, it makes the parsing much less verbose and also more intuitive, since the --addr is a feature of the kvs-client as opposed to the specific command being executed.

    Is your feature request related to a problem?

    No.

    Describe the feature you'd like

    Change the requirements for project 3 to indicate that the --addr field should show up in the beginning of the command. In other words, the specs should look something like this:

    USAGE:
        kvs-client [OPTIONS] <SUBCOMMAND>
    
    FLAGS:
        -h, --help       Prints help information
        -V, --version    Prints version information
    
    OPTIONS:
            --addr <addr>     [default: 127.0.0.1:4000]
    
    SUBCOMMANDS:
        get     
        help    Prints this message or the help of the given subcommand(s)
        rm      
        set     
    

    Describe alternatives you've considered

    There is no straightforward way (that I've come across in the past couple hours) that allows us to easily append a flag that applies to all subcommands at the very end.

    type/enhancement 
    opened by parthik19 0
  • TP201 Project3: write benchmark comparison between kvs and sled

    TP201 Project3: write benchmark comparison between kvs and sled

    General Question

    I am writing TP201 Project3. When doing write benchmark, I find that sled's set operation is much slower than kvs. Screen Shot 2021-01-09 at 12 06 15 AM

    I also run the Project3 code in this repo and find similar phenomenon. Screen Shot 2021-01-08 at 11 38 32 PM Write benchmark for sled takes so much time that it is not yet finished in the snapshot.

    Is it normal that sled's set operation is much slower than kvs or I made mistakes somewhere.

    Tips

    Expect for asking questions on Github, you can also discuss it in the channel #wg-talent-plan-courses. of tidbcommunity slack workspace.

    type/question 
    opened by xuyifangreeneyes 1
  • update tp102-how-to-use-git-github.md

    update tp102-how-to-use-git-github.md

    Bug Report

    Please answer these questions before submitting your issue. Thanks!

    1. What did you do?

    修复有个文本错误,这个错误导致 markdown 语法无法成功渲染文本

    2. What did you expect to see?

    3. What did you see instead?

    在文档中: 正确文本应当是:**Suggested Assignments:**,而非:**Suggested Assignments: ** 前者冒号后面多出来的空格,将导致文本无法被加粗

    type/bug 
    opened by callmePicacho 1
  • fix(courses/rust/project-2): BufWriterWithPos seek end of file

    fix(courses/rust/project-2): BufWriterWithPos seek end of file

    When opening a file with append, although writing the file starts from the end, the result of seeking the current position does start, and it needs to specifiy seek to the end of the file.

    opened by wutz 0
  • use and import cli.yml file to finish project-1 failed

    use and import cli.yml file to finish project-1 failed

    General Question

    The sourse in github using clap crate is coding in the kvs.rs file. However, I use the cli.yml to do it, and I can not pass some test cases.

    cli.yml file

    name: project-1
    version: "1.0"
    author: *
    about: get some args to do something
    args:
        - INPUT:
            help: the flag to input
            required: true
            index: 1
        - V:
            short: V
            multiple: false
            help: show the version
    subcommands:
        - set:
            about: to set the key-value with the input
            version: "1.0"
            author: *
            args:
                - key:
                    index: 1
                    help: the key is needed.
                    required: true
                - value:
                    index: 2
                    help: the value is needed.
                    required: true
        - get:
            about: to get the value by the key
            version: "1.0"
            author: *
            args:
                - key:
                    index: 1
                    help: the key is needed.
                    required: true
        - rm:
            about: to remove the key-value by the key
            version: "1.0"
            author: *
            args:
                - key:
                    index: 1
                    help: the key is needed.
                    required: true
        
    

    kvs.rs file

    #[macro_use]
    extern crate clap;
    use clap::App;
    use std::process::exit;
    
    fn main() {
    
        let yaml = load_yaml!("cli.yml");
        let matches = App::from_yaml(yaml).get_matches();
    
        if matches.is_present("V") {
            println!(env!("CARGO_PKG_VERSION"));
        }
    
        match matches.subcommand() {
            ("set", Some(_matches)) => {
                eprintln!("unimplemented");
                exit(1);
            }
            ("get", Some(_matches)) => {
                eprintln!("unimplemented");
                exit(1);
            }
            ("rm", Some(_matches)) => {
                eprintln!("unimplemented");
                exit(1);
            }
            _ => unreachable!(),
        }
    }
    

    the test result

    running 13 tests
    test cli_no_args ... ok
    test cli_invalid_subcommand ... ok
    test get_non_existent_value ... ok
    test cli_get ... FAILED
    test get_stored_value ... ok
    test overwrite_value ... ok
    test remove_key ... ok
    test cli_rm ... FAILED
    test cli_set ... FAILED
    test cli_invalid_get ... ok
    test cli_invalid_rm ... ok
    test cli_version ... FAILED
    test cli_invalid_set ... ok
    
    type/question 
    opened by PhzCode 0
  • Chinese Translation

    Chinese Translation

    Is there a Chinese version of the documents? If not, do you mind my translating them into Chinese?

    type/enhancement 
    opened by rhanqtl 0
  • Update README.md

    Update README.md

    What problem does this PR solve?

    • Issue number: close #371
    • Brief description of the problem: grammatical errors, typos, punctuation, Chinglish, content inconsistency.

    What is changed and how it works?

    • Polished language.
    • Added information below heading.
    • Corrected grammatical errors and typos.

    Check List

    see issue #371

    Side effects

    • misunderstanding
    • decreased readability
    opened by The-rambunctious-S 0
Owner
PingCAP
The team behind TiDB TiKV, an open source MySQL compatible NewSQL HTAP database
PingCAP
Type safe SQL builder with code generation and automatic query result data mapping

Jet Jet is a complete solution for efficient and high performance database access, consisting of type-safe SQL builder with code generation and automa

null 482 Sep 21, 2021
Mocking your SQL database in Go tests has never been easier.

copyist Mocking your SQL database in Go tests has never been easier. The copyist library automatically records low-level SQL calls made during your te

CockroachDB 772 Sep 14, 2021
A Go (golang) package that enhances the standard database/sql package by providing powerful data retrieval methods as well as DB-agnostic query building capabilities.

ozzo-dbx Summary Description Requirements Installation Supported Databases Getting Started Connecting to Database Executing Queries Binding Parameters

Ozzo Framework 541 Aug 26, 2021
Document-oriented, embedded SQL database

Genji Document-oriented, embedded, SQL database Table of contents Table of contents Introduction Features Installation Usage Using Genji's API Using d

Genji 734 Sep 19, 2021
Type safe SQL query builder and struct mapper for Go

sq (Structured Query) ?? ?? sq is a code-generated, type safe query builder and struct mapper for Go. ?? ?? Documentation • Reference • Examples This

null 123 Sep 13, 2021
GraphJin - Build APIs in 5 minutes with GraphQL. An instant GraphQL to SQL compiler.

GraphJin - Build APIs in 5 minutes GraphJin gives you a high performance GraphQL API without you having to write any code. GraphQL is automagically co

Vikram Rangnekar 1.5k Sep 25, 2021
Go database query builder library for PostgreSQL

buildsqlx Go Database query builder library Installation Selects, Ordering, Limit & Offset GroupBy / Having Where, AndWhere, OrWhere clauses WhereIn /

Arthur Kushman 38 Sep 2, 2021
Go library for accessing multi-host SQL database installations

hasql hasql provides simple and reliable way to access high-availability database setups with multiple hosts. Status hasql is production-ready and is

Yandex 84 Sep 7, 2021
LBADD: An experimental, distributed SQL database

LBADD Let's build a distributed database. LBADD is an experimental distributed SQL database, written in Go. The goal of this project is to build a dat

Tom Arrell 375 Sep 12, 2021
Additions to Go's database/sql for super fast performance and convenience. (fork of gocraft/dbr)

dbr (fork of gocraft/dbr) provides additions to Go's database/sql for super fast performance and convenience. Getting Started // create a connection (

Free and open source software developed at Mail.Ru 148 Sep 22, 2021
💥 A lightweight DSL & ORM which helps you to write SQL in Go.

sqlingo is a SQL DSL (a.k.a. SQL Builder or ORM) library in Go. It generates code from the database and lets you write SQL queries in an elegant way.

Qishuai Liu 156 Sep 17, 2021
igor is an abstraction layer for PostgreSQL with a gorm like syntax.

igor igor is an abstraction layer for PostgreSQL, written in Go. Igor syntax is (almost) compatible with GORM. When to use igor You should use igor wh

Paolo Galeone 84 Aug 2, 2021
Database Abstraction Layer (dbal) for Go. Support SQL builder and get result easily (now only support mysql)

godbal Database Abstraction Layer (dbal) for go (now only support mysql) Motivation I wanted a DBAL that No ORM、No Reflect、Concurrency Save, support S

徐佳军 52 Apr 22, 2021
An early PostgreSQL implementation in Go

gosql An early PostgreSQL implementation in Go. Example $ git clone [email protected]:eatonphil/gosql $ cd gosql $ go run cmd/main.go Welcome to gosql. #

Phil Eaton 526 Sep 14, 2021
Zero boilerplate database operations for Go

(Now compatible with MySQL and PostgreSQL!) Everyone knows that performing simple DATABASE queries in Go takes numerous lines of code that is often re

null 317 Sep 17, 2021
A Go library for collecting sql.DBStats in Prometheus format

sqlstats A Go library for collecting sql.DBStats and exporting them in Prometheus format. A sql.DB object represents a pool of zero or more underlying

Daniel Middlecote 128 Aug 8, 2021
SQL builder and query library for golang

__ _ ___ __ _ _ _ / _` |/ _ \ / _` | | | | | (_| | (_) | (_| | |_| | \__, |\___/ \__, |\__,_| |___/ |_| goqu is an expressive SQL bu

Doug Martin 1.3k Sep 23, 2021
golang orm and sql builder

gosql gosql is a easy ORM library for Golang. Style: var userList []UserModel err := db.FetchAll(&userList, gosql.Columns("id","name"), gosql.

RushTeam 147 Sep 23, 2021
Simple SQL extensions for Go

go-sx provides some extensions to the standard library database/sql package. It is designed for those who wish to use the full power of SQL without a heavy abstraction layer.

travel audience 178 Aug 24, 2021