mirror of
https://github.com/openimsdk/open-im-server.git
synced 2026-05-18 07:49:01 +08:00
feat: Remove MySQL and implement it all with Mongo (#1508)
* fix: GetUserReqApplicationList error when there is a disbanded group chat
* fix: error when querying some information about disbanded group
* fix: GetUserReqApplicationList dismissed group error
* fix: the original message referenced by the pull message processing is withdrawn
* fix: the original message referenced by the pull message processing is withdrawn
* fix: the original message referenced by the pull message processing is withdrawn
* fix: the original message referenced by the pull message processing is withdrawn
* fix: the original message referenced by the pull message processing is withdrawn
* fix: the original message referenced by the pull message processing is withdrawn
* fix: the original message referenced by the pull message processing is withdrawn
* fix: the original message referenced by the pull message processing is withdrawn
* fix: the original message referenced by the pull message processing is withdrawn
* merge
* cicd: robot automated Change
* sdkws.MsgData
* user
* interface{} -> any
* user
* third
* group
* group
* group
* group
* group
* group
* conversation
* standalone mysql db model
* tx
* s3
* group
* mongo
* group
* group
* group
* group
* group
* group
* refactor: add openim mysql to mongo refactor
Signed-off-by: Xinwei Xiong(cubxxw) <3293172751nss@gmail.com>
* refactor: add openim mysql to mongo refactor
Signed-off-by: Xinwei Xiong(cubxxw) <3293172751nss@gmail.com>
* remove mysql
* remove mysql
* friend
* friend
* friend
* friend
* friend
* friend
* group
* convert
* index
* index
* all
* all
* mysql2mongo
* data conversion
* up35
* up35
* feat: add format set
Signed-off-by: Xinwei Xiong(cubxxw) <3293172751nss@gmail.com>
* fix: fix scripts
Signed-off-by: Xinwei Xiong(cubxxw) <3293172751nss@gmail.com>
* merge main
* merge main
* Update init-config.sh
* fix: user args check
---------
Signed-off-by: Xinwei Xiong(cubxxw) <3293172751nss@gmail.com>
Co-authored-by: withchao <withchao@users.noreply.github.com>
Co-authored-by: Xinwei Xiong(cubxxw) <3293172751nss@gmail.com>
Co-authored-by: Xinwei Xiong <3293172751@qq.com>
This commit is contained in:
chao
@@ -0,0 +1,67 @@
|
||||
# README for OpenIM Server Data Conversion Tool
|
||||
|
||||
## Overview
|
||||
|
||||
This tool is part of the OpenIM Server suite, specifically designed for data conversion between MySQL and MongoDB databases. It handles the migration of various data types, including user information, friendships, group memberships, and more from a MySQL database to MongoDB, ensuring data consistency and integrity during the transition.
|
||||
|
||||
## Features
|
||||
|
||||
+ **Configurable Database Connections:** Supports connections to both MySQL and MongoDB, configurable through a YAML file.
|
||||
+ **Data Conversion Tasks:** Converts a range of data models, including user profiles, friend requests, group memberships, and logs.
|
||||
+ **Version Control:** Maintains data versioning, ensuring only necessary migrations are performed.
|
||||
+ **Error Handling:** Robust error handling for database connectivity and query execution.
|
||||
|
||||
## Requirements
|
||||
|
||||
+ Go programming environment
|
||||
+ MySQL and MongoDB servers
|
||||
+ OpenIM Server dependencies installed
|
||||
|
||||
## Installation
|
||||
|
||||
1. Ensure Go is installed and set up on your system.
|
||||
2. Clone the OpenIM Server repository.
|
||||
3. Navigate to the directory containing this tool.
|
||||
4. Install required dependencies.
|
||||
|
||||
## Configuration
|
||||
|
||||
+ Configuration is managed through a YAML file specified at runtime.
|
||||
+ Set up the MySQL and MongoDB connection parameters in the config file.
|
||||
|
||||
## Usage
|
||||
|
||||
To run the tool, use the following command from the terminal:
|
||||
|
||||
```go
|
||||
make build BINS="up35"
|
||||
```
|
||||
|
||||
Where `path/to/config.yaml` is the path to your configuration file.
|
||||
|
||||
## Functionality
|
||||
|
||||
The main functions of the script include:
|
||||
|
||||
+ `InitConfig(path string)`: Reads and parses the YAML configuration file.
|
||||
+ `GetMysql()`: Establishes a connection to the MySQL database.
|
||||
+ `GetMongo()`: Establishes a connection to the MongoDB database.
|
||||
+ `Main(path string)`: Orchestrates the data migration process.
|
||||
+ `SetMongoDataVersion(db *mongo.Database, curver string)`: Updates the data version in MongoDB after migration.
|
||||
+ `NewTask(...)`: Generic function to handle the migration of different data types.
|
||||
+ `insertMany(coll *mongo.Collection, objs []any)`: Inserts multiple records into a MongoDB collection.
|
||||
+ `getColl(obj any)`: Retrieves the MongoDB collection associated with a given object.
|
||||
+ `convert struct`: Contains methods for converting MySQL models to MongoDB models.
|
||||
|
||||
## Notes
|
||||
|
||||
+ Ensure that the MySQL and MongoDB instances are accessible and that the credentials provided in the config file are correct.
|
||||
+ It is advisable to backup databases before running the migration to prevent data loss.
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions to improve the tool or address issues are welcome. Please follow the project's contribution guidelines.
|
||||
|
||||
## License
|
||||
|
||||
Refer to the project's license document for usage and distribution rights.
|
||||
@@ -0,0 +1,51 @@
|
||||
module github.com/openimsdk/open-im-server/v3/tools/up35
|
||||
|
||||
go 1.19
|
||||
|
||||
require (
|
||||
github.com/go-sql-driver/mysql v1.7.1
|
||||
github.com/openimsdk/open-im-server/v3 v3.5.0
|
||||
github.com/openimsdk/open-im-server/v3/tools/data-conversion v0.0.0-00010101000000-000000000000
|
||||
go.mongodb.org/mongo-driver v1.12.1
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
gorm.io/driver/mysql v1.5.1
|
||||
gorm.io/gorm v1.25.4
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/OpenIMSDK/protocol v0.0.31 // indirect
|
||||
github.com/OpenIMSDK/tools v0.0.18 // indirect
|
||||
github.com/bwmarrin/snowflake v0.3.0 // indirect
|
||||
github.com/golang/protobuf v1.5.3 // indirect
|
||||
github.com/golang/snappy v0.0.4 // indirect
|
||||
github.com/jinzhu/copier v0.4.0 // indirect
|
||||
github.com/jinzhu/inflection v1.0.0 // indirect
|
||||
github.com/jinzhu/now v1.1.5 // indirect
|
||||
github.com/klauspost/compress v1.16.7 // indirect
|
||||
github.com/lestrrat-go/file-rotatelogs v2.4.0+incompatible // indirect
|
||||
github.com/lestrrat-go/strftime v1.0.6 // indirect
|
||||
github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe // indirect
|
||||
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646 // indirect
|
||||
github.com/pkg/errors v0.9.1 // indirect
|
||||
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
|
||||
github.com/xdg-go/scram v1.1.2 // indirect
|
||||
github.com/xdg-go/stringprep v1.0.4 // indirect
|
||||
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect
|
||||
go.uber.org/atomic v1.7.0 // indirect
|
||||
go.uber.org/multierr v1.6.0 // indirect
|
||||
go.uber.org/zap v1.24.0 // indirect
|
||||
golang.org/x/crypto v0.14.0 // indirect
|
||||
golang.org/x/image v0.13.0 // indirect
|
||||
golang.org/x/net v0.17.0 // indirect
|
||||
golang.org/x/sync v0.4.0 // indirect
|
||||
golang.org/x/sys v0.14.0 // indirect
|
||||
golang.org/x/text v0.13.0 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20231012201019-e917dd12ba7a // indirect
|
||||
google.golang.org/grpc v1.59.0 // indirect
|
||||
google.golang.org/protobuf v1.31.0 // indirect
|
||||
)
|
||||
|
||||
replace (
|
||||
github.com/openimsdk/open-im-server/v3 => ./../../../open-im-server
|
||||
github.com/openimsdk/open-im-server/v3/tools/data-conversion => ./../data-conversion
|
||||
)
|
||||
@@ -0,0 +1,125 @@
|
||||
github.com/OpenIMSDK/protocol v0.0.31 h1:ax43x9aqA6EKNXNukS5MT5BSTqkUmwO4uTvbJLtzCgE=
|
||||
github.com/OpenIMSDK/protocol v0.0.31/go.mod h1:F25dFrwrIx3lkNoiuf6FkCfxuwf8L4Z8UIsdTHP/r0Y=
|
||||
github.com/OpenIMSDK/tools v0.0.18 h1:h3CvKB90DNd2aIJcOQ99cqgeW6C0na0PzR1TNsfxwL0=
|
||||
github.com/OpenIMSDK/tools v0.0.18/go.mod h1:eg+q4A34Qmu73xkY0mt37FHGMCMfC6CtmOnm0kFEGFI=
|
||||
github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8=
|
||||
github.com/bwmarrin/snowflake v0.3.0 h1:xm67bEhkKh6ij1790JB83OujPR5CzNe8QuQqAgISZN0=
|
||||
github.com/bwmarrin/snowflake v0.3.0/go.mod h1:NdZxfVWX+oR6y2K0o6qAYv6gIOP9rjG0/E9WsDpxqwE=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/go-sql-driver/mysql v1.7.0/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI=
|
||||
github.com/go-sql-driver/mysql v1.7.1 h1:lUIinVbN1DY0xBg0eMOzmmtGoHwWBbvnWubQUrtU8EI=
|
||||
github.com/go-sql-driver/mysql v1.7.1/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI=
|
||||
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
|
||||
github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
|
||||
github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
|
||||
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
|
||||
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||
github.com/jinzhu/copier v0.4.0 h1:w3ciUoD19shMCRargcpm0cm91ytaBhDvuRpz1ODO/U8=
|
||||
github.com/jinzhu/copier v0.4.0/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg=
|
||||
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
|
||||
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
|
||||
github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
|
||||
github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
|
||||
github.com/jonboulle/clockwork v0.4.0 h1:p4Cf1aMWXnXAUh8lVfewRBx1zaTSYKrKMF2g3ST4RZ4=
|
||||
github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
|
||||
github.com/klauspost/compress v1.16.7 h1:2mk3MPGNzKyxErAw8YaohYh69+pa4sIQSC0fPGCFR9I=
|
||||
github.com/klauspost/compress v1.16.7/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
|
||||
github.com/lestrrat-go/envload v0.0.0-20180220234015-a3eb8ddeffcc h1:RKf14vYWi2ttpEmkA4aQ3j4u9dStX2t4M8UM6qqNsG8=
|
||||
github.com/lestrrat-go/envload v0.0.0-20180220234015-a3eb8ddeffcc/go.mod h1:kopuH9ugFRkIXf3YoqHKyrJ9YfUFsckUU9S7B+XP+is=
|
||||
github.com/lestrrat-go/file-rotatelogs v2.4.0+incompatible h1:Y6sqxHMyB1D2YSzWkLibYKgg+SwmyFU9dF2hn6MdTj4=
|
||||
github.com/lestrrat-go/file-rotatelogs v2.4.0+incompatible/go.mod h1:ZQnN8lSECaebrkQytbHj4xNgtg8CR7RYXnPok8e0EHA=
|
||||
github.com/lestrrat-go/strftime v1.0.6 h1:CFGsDEt1pOpFNU+TJB0nhz9jl+K0hZSLE205AhTIGQQ=
|
||||
github.com/lestrrat-go/strftime v1.0.6/go.mod h1:f7jQKgV5nnJpYgdEasS+/y7EsTb8ykN2z68n3TtcTaw=
|
||||
github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe h1:iruDEfMl2E6fbMZ9s0scYfZQ84/6SPL6zC8ACM2oIL0=
|
||||
github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
|
||||
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646 h1:zYyBkD/k9seD2A7fsi6Oo2LfFZAehjjQMERAvZLEDnQ=
|
||||
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646/go.mod h1:jpp1/29i3P1S/RLdc7JQKbRpFeM1dOBd8T9ki5s+AY8=
|
||||
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
|
||||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
|
||||
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
|
||||
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
|
||||
github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
|
||||
github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
|
||||
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
|
||||
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
|
||||
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d h1:splanxYIlg+5LfHAM6xpdFEAYOk8iySO56hMFq6uLyA=
|
||||
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
|
||||
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||
go.mongodb.org/mongo-driver v1.12.1 h1:nLkghSU8fQNaK7oUmDhQFsnrtcoNy7Z6LVFKsEecqgE=
|
||||
go.mongodb.org/mongo-driver v1.12.1/go.mod h1:/rGBTebI3XYboVmgz+Wv3Bcbl3aD0QF9zl6kDDw18rQ=
|
||||
go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw=
|
||||
go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
|
||||
go.uber.org/goleak v1.1.11 h1:wy28qYRKZgnJTxGxvye5/wgWr1EKjmUDGYox5mGlRlI=
|
||||
go.uber.org/multierr v1.6.0 h1:y6IPFStTAIT5Ytl7/XYmHvzXQ7S3g/IeZW9hyZ5thw4=
|
||||
go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
|
||||
go.uber.org/zap v1.24.0 h1:FiJd5l1UOLj0wCgbSE0rwwXHzEdAZS6hiiSnxJN/D60=
|
||||
go.uber.org/zap v1.24.0/go.mod h1:2kMP+WWQ8aoFoedH3T2sq6iJ2yDWpHbP0f6MQbS9Gkg=
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
|
||||
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
|
||||
golang.org/x/crypto v0.14.0 h1:wBqGXzWJW6m1XrIKlAH0Hs1JJ7+9KBwnIO8v66Q9cHc=
|
||||
golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
|
||||
golang.org/x/image v0.13.0 h1:3cge/F/QTkNLauhf2QoE9zp+7sr+ZcL4HnoZmdwg9sg=
|
||||
golang.org/x/image v0.13.0/go.mod h1:6mmbMOeV28HuMTgA6OSRkdXKYw/t5W9Uwn2Yv1r3Yxk=
|
||||
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
|
||||
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
|
||||
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
|
||||
golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
|
||||
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
|
||||
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.4.0 h1:zxkM55ReGkDlKSM+Fu41A+zmbZuaPVbGMzvvdUPznYQ=
|
||||
golang.org/x/sync v0.4.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.14.0 h1:Vz7Qs629MkJkGyHxUlRHizWJRG2j8fbQKjELVSNhy7Q=
|
||||
golang.org/x/sys v0.14.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
||||
golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
|
||||
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
|
||||
golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k=
|
||||
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
|
||||
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
|
||||
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20231012201019-e917dd12ba7a h1:a2MQQVoTo96JC9PMGtGBymLp7+/RzpFc2yX/9WfFg1c=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20231012201019-e917dd12ba7a/go.mod h1:4cYg8o5yUbm77w8ZX00LhMVNl/YVBFJRYWDc0uYWMs0=
|
||||
google.golang.org/grpc v1.59.0 h1:Z5Iec2pjwb+LEOqzpB2MR12/eKFhDPhuqW91O+4bwUk=
|
||||
google.golang.org/grpc v1.59.0/go.mod h1:aUPDwccQo6OTjy7Hct4AfBPD1GptF4fyUjIkQ9YtF98=
|
||||
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
|
||||
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
|
||||
google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8=
|
||||
google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gorm.io/driver/mysql v1.5.1 h1:WUEH5VF9obL/lTtzjmML/5e6VfFR/788coz2uaVCAZw=
|
||||
gorm.io/driver/mysql v1.5.1/go.mod h1:Jo3Xu7mMhCyj8dlrb3WoCaRd1FhsVh+yMXb1jUInf5o=
|
||||
gorm.io/gorm v1.25.1/go.mod h1:L4uxeKpfBml98NYqVqwAdmV1a2nBtAec/cf3fpucW/k=
|
||||
gorm.io/gorm v1.25.4 h1:iyNd8fNAe8W9dvtlgeRI5zSVZPsq3OpcTu37cYcpCmw=
|
||||
gorm.io/gorm v1.25.4/go.mod h1:L4uxeKpfBml98NYqVqwAdmV1a2nBtAec/cf3fpucW/k=
|
||||
@@ -0,0 +1,367 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"github.com/go-sql-driver/mysql"
|
||||
"github.com/openimsdk/open-im-server/v3/pkg/common/config"
|
||||
"github.com/openimsdk/open-im-server/v3/pkg/common/db/mgo"
|
||||
mongoModel "github.com/openimsdk/open-im-server/v3/pkg/common/db/table/relation"
|
||||
"github.com/openimsdk/open-im-server/v3/pkg/common/db/unrelation"
|
||||
mysqlModel "github.com/openimsdk/open-im-server/v3/tools/data-conversion/openim/mysql/v3"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
"gopkg.in/yaml.v3"
|
||||
gormMysql "gorm.io/driver/mysql"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
"log"
|
||||
"os"
|
||||
"reflect"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
const (
|
||||
versionTable = "dataver"
|
||||
versionKey = "data_version"
|
||||
versionValue = 35
|
||||
)
|
||||
|
||||
func main() {
|
||||
var path string
|
||||
flag.StringVar(&path, "c", "", "path config file")
|
||||
flag.Parse()
|
||||
if err := Main(path); err != nil {
|
||||
log.Fatal(err)
|
||||
return
|
||||
}
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
func InitConfig(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return yaml.Unmarshal(data, &config.Config)
|
||||
}
|
||||
|
||||
func GetMysql() (*gorm.DB, error) {
|
||||
conf := config.Config.Mysql
|
||||
mysqlDSN := fmt.Sprintf("%s:%s@tcp(%s)/%s?charset=utf8mb4&parseTime=True&loc=Local", conf.Username, conf.Password, conf.Address[0], conf.Database)
|
||||
return gorm.Open(gormMysql.Open(mysqlDSN), &gorm.Config{Logger: logger.Discard})
|
||||
}
|
||||
|
||||
func GetMongo() (*mongo.Database, error) {
|
||||
mgo, err := unrelation.NewMongo()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return mgo.GetDatabase(), nil
|
||||
}
|
||||
|
||||
func Main(path string) error {
|
||||
if err := InitConfig(path); err != nil {
|
||||
return err
|
||||
}
|
||||
if config.Config.Mysql == nil {
|
||||
return nil
|
||||
}
|
||||
mongoDB, err := GetMongo()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var version struct {
|
||||
Key string `bson:"key"`
|
||||
Value string `bson:"value"`
|
||||
}
|
||||
switch mongoDB.Collection(versionTable).FindOne(context.Background(), bson.M{"key": versionKey}).Decode(&version) {
|
||||
case nil:
|
||||
if ver, _ := strconv.Atoi(version.Value); ver >= versionValue {
|
||||
return nil
|
||||
}
|
||||
case mongo.ErrNoDocuments:
|
||||
default:
|
||||
return err
|
||||
}
|
||||
mysqlDB, err := GetMysql()
|
||||
if err != nil {
|
||||
if mysqlErr, ok := err.(*mysql.MySQLError); ok && mysqlErr.Number == 1049 {
|
||||
if err := SetMongoDataVersion(mongoDB, version.Value); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil // database not exist
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
var c convert
|
||||
var tasks []func() error
|
||||
tasks = append(tasks,
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewUserMongo, c.User) },
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewFriendMongo, c.Friend) },
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewFriendRequestMongo, c.FriendRequest) },
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewBlackMongo, c.Black) },
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewGroupMongo, c.Group) },
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewGroupMember, c.GroupMember) },
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewGroupRequestMgo, c.GroupRequest) },
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewConversationMongo, c.Conversation) },
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewS3Mongo, c.Object(config.Config.Object.Enable)) },
|
||||
func() error { return NewTask(mysqlDB, mongoDB, mgo.NewLogMongo, c.Log) },
|
||||
)
|
||||
|
||||
for _, task := range tasks {
|
||||
if err := task(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if err := SetMongoDataVersion(mongoDB, version.Value); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func SetMongoDataVersion(db *mongo.Database, curver string) error {
|
||||
filter := bson.M{"key": versionKey, "value": curver}
|
||||
update := bson.M{"$set": bson.M{"key": versionKey, "value": strconv.Itoa(versionValue)}}
|
||||
_, err := db.Collection(versionTable).UpdateOne(context.Background(), filter, update, options.Update().SetUpsert(true))
|
||||
return err
|
||||
}
|
||||
|
||||
// NewTask A mysql table B mongodb model C mongodb table
|
||||
func NewTask[A interface{ TableName() string }, B any, C any](gormDB *gorm.DB, mongoDB *mongo.Database, mongoDBInit func(db *mongo.Database) (B, error), convert func(v A) C) error {
|
||||
obj, err := mongoDBInit(mongoDB)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var zero A
|
||||
tableName := zero.TableName()
|
||||
coll, err := getColl(obj)
|
||||
if err != nil {
|
||||
return fmt.Errorf("get mongo collection %s failed, err: %w", tableName, err)
|
||||
}
|
||||
var count int
|
||||
defer func() {
|
||||
log.Printf("completed convert %s total %d\n", tableName, count)
|
||||
}()
|
||||
const batch = 100
|
||||
for page := 0; ; page++ {
|
||||
res := make([]A, 0, batch)
|
||||
if err := gormDB.Limit(batch).Offset(page * batch).Find(&res).Error; err != nil {
|
||||
if mysqlErr, ok := err.(*mysql.MySQLError); ok && mysqlErr.Number == 1146 {
|
||||
return nil // table not exist
|
||||
}
|
||||
return fmt.Errorf("find mysql table %s failed, err: %w", tableName, err)
|
||||
}
|
||||
if len(res) == 0 {
|
||||
return nil
|
||||
}
|
||||
temp := make([]any, len(res))
|
||||
for i := range res {
|
||||
temp[i] = convert(res[i])
|
||||
}
|
||||
if err := insertMany(coll, temp); err != nil {
|
||||
return fmt.Errorf("insert mongo table %s failed, err: %w", tableName, err)
|
||||
}
|
||||
count += len(res)
|
||||
if len(res) < batch {
|
||||
return nil
|
||||
}
|
||||
log.Printf("current convert %s completed %d\n", tableName, count)
|
||||
}
|
||||
}
|
||||
|
||||
func insertMany(coll *mongo.Collection, objs []any) error {
|
||||
if _, err := coll.InsertMany(context.Background(), objs); err != nil {
|
||||
if !mongo.IsDuplicateKeyError(err) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for i := range objs {
|
||||
_, err := coll.InsertOne(context.Background(), objs[i])
|
||||
switch {
|
||||
case err == nil:
|
||||
case mongo.IsDuplicateKeyError(err):
|
||||
default:
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func getColl(obj any) (_ *mongo.Collection, err error) {
|
||||
defer func() {
|
||||
if e := recover(); e != nil {
|
||||
err = fmt.Errorf("not found %+v", e)
|
||||
}
|
||||
}()
|
||||
stu := reflect.ValueOf(obj).Elem()
|
||||
typ := reflect.TypeOf(&mongo.Collection{}).String()
|
||||
for i := 0; i < stu.NumField(); i++ {
|
||||
field := stu.Field(i)
|
||||
if field.Type().String() == typ {
|
||||
return (*mongo.Collection)(field.UnsafePointer()), nil
|
||||
}
|
||||
}
|
||||
return nil, errors.New("not found")
|
||||
}
|
||||
|
||||
type convert struct{}
|
||||
|
||||
func (convert) User(v mysqlModel.UserModel) mongoModel.UserModel {
|
||||
return mongoModel.UserModel{
|
||||
UserID: v.UserID,
|
||||
Nickname: v.Nickname,
|
||||
FaceURL: v.FaceURL,
|
||||
Ex: v.Ex,
|
||||
AppMangerLevel: v.AppMangerLevel,
|
||||
GlobalRecvMsgOpt: v.GlobalRecvMsgOpt,
|
||||
CreateTime: v.CreateTime,
|
||||
}
|
||||
}
|
||||
|
||||
func (convert) Friend(v mysqlModel.FriendModel) mongoModel.FriendModel {
|
||||
return mongoModel.FriendModel{
|
||||
OwnerUserID: v.OwnerUserID,
|
||||
FriendUserID: v.FriendUserID,
|
||||
Remark: v.Remark,
|
||||
CreateTime: v.CreateTime,
|
||||
AddSource: v.AddSource,
|
||||
OperatorUserID: v.OperatorUserID,
|
||||
Ex: v.Ex,
|
||||
}
|
||||
}
|
||||
|
||||
func (convert) FriendRequest(v mysqlModel.FriendRequestModel) mongoModel.FriendRequestModel {
|
||||
return mongoModel.FriendRequestModel{
|
||||
FromUserID: v.FromUserID,
|
||||
ToUserID: v.ToUserID,
|
||||
HandleResult: v.HandleResult,
|
||||
ReqMsg: v.ReqMsg,
|
||||
CreateTime: v.CreateTime,
|
||||
HandlerUserID: v.HandlerUserID,
|
||||
HandleMsg: v.HandleMsg,
|
||||
HandleTime: v.HandleTime,
|
||||
Ex: v.Ex,
|
||||
}
|
||||
}
|
||||
|
||||
func (convert) Black(v mysqlModel.BlackModel) mongoModel.BlackModel {
|
||||
return mongoModel.BlackModel{
|
||||
OwnerUserID: v.OwnerUserID,
|
||||
BlockUserID: v.BlockUserID,
|
||||
CreateTime: v.CreateTime,
|
||||
AddSource: v.AddSource,
|
||||
OperatorUserID: v.OperatorUserID,
|
||||
Ex: v.Ex,
|
||||
}
|
||||
}
|
||||
|
||||
func (convert) Group(v mysqlModel.GroupModel) mongoModel.GroupModel {
|
||||
return mongoModel.GroupModel{
|
||||
GroupID: v.GroupID,
|
||||
GroupName: v.GroupName,
|
||||
Notification: v.Notification,
|
||||
Introduction: v.Introduction,
|
||||
FaceURL: v.FaceURL,
|
||||
CreateTime: v.CreateTime,
|
||||
Ex: v.Ex,
|
||||
Status: v.Status,
|
||||
CreatorUserID: v.CreatorUserID,
|
||||
GroupType: v.GroupType,
|
||||
NeedVerification: v.NeedVerification,
|
||||
LookMemberInfo: v.LookMemberInfo,
|
||||
ApplyMemberFriend: v.ApplyMemberFriend,
|
||||
NotificationUpdateTime: v.NotificationUpdateTime,
|
||||
NotificationUserID: v.NotificationUserID,
|
||||
}
|
||||
}
|
||||
|
||||
func (convert) GroupMember(v mysqlModel.GroupMemberModel) mongoModel.GroupMemberModel {
|
||||
return mongoModel.GroupMemberModel{
|
||||
GroupID: v.GroupID,
|
||||
UserID: v.UserID,
|
||||
Nickname: v.Nickname,
|
||||
FaceURL: v.FaceURL,
|
||||
RoleLevel: v.RoleLevel,
|
||||
JoinTime: v.JoinTime,
|
||||
JoinSource: v.JoinSource,
|
||||
InviterUserID: v.InviterUserID,
|
||||
OperatorUserID: v.OperatorUserID,
|
||||
MuteEndTime: v.MuteEndTime,
|
||||
Ex: v.Ex,
|
||||
}
|
||||
}
|
||||
|
||||
func (convert) GroupRequest(v mysqlModel.GroupRequestModel) mongoModel.GroupRequestModel {
|
||||
return mongoModel.GroupRequestModel{
|
||||
UserID: v.UserID,
|
||||
GroupID: v.GroupID,
|
||||
HandleResult: v.HandleResult,
|
||||
ReqMsg: v.ReqMsg,
|
||||
HandledMsg: v.HandledMsg,
|
||||
ReqTime: v.ReqTime,
|
||||
HandleUserID: v.HandleUserID,
|
||||
HandledTime: v.HandledTime,
|
||||
JoinSource: v.JoinSource,
|
||||
InviterUserID: v.InviterUserID,
|
||||
Ex: v.Ex,
|
||||
}
|
||||
}
|
||||
|
||||
func (convert) Conversation(v mysqlModel.ConversationModel) mongoModel.ConversationModel {
|
||||
return mongoModel.ConversationModel{
|
||||
OwnerUserID: v.OwnerUserID,
|
||||
ConversationID: v.ConversationID,
|
||||
ConversationType: v.ConversationType,
|
||||
UserID: v.UserID,
|
||||
GroupID: v.GroupID,
|
||||
RecvMsgOpt: v.RecvMsgOpt,
|
||||
IsPinned: v.IsPinned,
|
||||
IsPrivateChat: v.IsPrivateChat,
|
||||
BurnDuration: v.BurnDuration,
|
||||
GroupAtType: v.GroupAtType,
|
||||
AttachedInfo: v.AttachedInfo,
|
||||
Ex: v.Ex,
|
||||
MaxSeq: v.MaxSeq,
|
||||
MinSeq: v.MinSeq,
|
||||
CreateTime: v.CreateTime,
|
||||
IsMsgDestruct: v.IsMsgDestruct,
|
||||
MsgDestructTime: v.MsgDestructTime,
|
||||
LatestMsgDestructTime: v.LatestMsgDestructTime,
|
||||
}
|
||||
}
|
||||
|
||||
func (convert) Object(engine string) func(v mysqlModel.ObjectModel) mongoModel.ObjectModel {
|
||||
return func(v mysqlModel.ObjectModel) mongoModel.ObjectModel {
|
||||
return mongoModel.ObjectModel{
|
||||
Name: v.Name,
|
||||
UserID: v.UserID,
|
||||
Hash: v.Hash,
|
||||
Engine: engine,
|
||||
Key: v.Key,
|
||||
Size: v.Size,
|
||||
ContentType: v.ContentType,
|
||||
Group: v.Cause,
|
||||
CreateTime: v.CreateTime,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (convert) Log(v mysqlModel.Log) mongoModel.LogModel {
|
||||
return mongoModel.LogModel{
|
||||
LogID: v.LogID,
|
||||
Platform: v.Platform,
|
||||
UserID: v.UserID,
|
||||
CreateTime: v.CreateTime,
|
||||
Url: v.Url,
|
||||
FileName: v.FileName,
|
||||
SystemType: v.SystemType,
|
||||
Version: v.Version,
|
||||
Ex: v.Ex,
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user