Migration (#120)

Go mod tidy

[skip ci] Add modules/migration to docs

[skip ci] update date

fmt

Merge branch 'master' into feature/migration

# Conflicts:
#	pkg/routes/api/v1/info.go

Add docs on how to create a migrator

Add available migrators to /info endpoint

Return a message once everything was migrated successfully

Add swagger docs for wunderlist migration

Docs for migration [skip ci]

Fix due date fixture in migration test

Fix staticcheck

Fix lint

Logging and cleanup

Make the migrator work with real data

Add routes for migration

Fix misspell

Add method to store a full vikunja structure into vikunja

Add getting all data from wunderlist

Add attachment migration from wunderlist

Add done and done at to wunderlist migrator

Add todo

Add wunderlist auth url implementation

Fix lint

Finish wunderlist migration

Added all structs for the wunderlist migratior

Fix owner field being null for user shared namespaces (#119)

Update copyright year (#118)

Add option to disable registration (#117)

Added migrator interface

Co-authored-by: kolaente <k@knt.li>
Reviewed-on: https://kolaente.dev/vikunja/api/pulls/120
This commit is contained in:
konrad 2020-01-19 16:52:16 +00:00
parent 1d2cdf0fb8
commit 9e39399689
19 changed files with 1451 additions and 11 deletions

View file

@ -117,3 +117,19 @@ files:
# The maximum size of a file, as a human-readable string. # The maximum size of a file, as a human-readable string.
# Warning: The max size is limited 2^64-1 bytes due to the underlying datatype # Warning: The max size is limited 2^64-1 bytes due to the underlying datatype
maxsize: 20MB maxsize: 20MB
migration:
# These are the settings for the wunderlist migrator
wunderlist:
# Wheter to enable the wunderlist migrator or not
enable: true
# The client id, required for making requests to the wunderlist api
# You need to register your vikunja instance at https://developer.wunderlist.com/apps/new to get this
clientid:
# The client secret, also required for making requests to the wunderlist api
clientsecret:
# The url where clients are redirected after they authorized Vikunja to access their wunderlist stuff.
# This needs to match the url you entered when registering your Vikunja instance at wunderlist.
# This is usually the frontend url where the frontend then makes a request to /migration/wunderlist/migrate
# with the code obtained from the wunderlist api.
redirecturl:

View file

@ -0,0 +1,91 @@
---
date: "2020-01-19:16:00+02:00"
title: "Migrations"
draft: false
type: "doc"
menu:
sidebar:
parent: "development"
---
# Writing a migrator for Vikunja
It is possible to migrate data from other to-do services to Vikunja.
To make this easier, we have put together a few helpers which are documented on this page.
In general, each migrator implements a migrator interface which is then called from a client.
The interface makes it possible to use helper methods which handle http an focus only on the implementation of the migrator itself.
### Structure
All migrator implementations live in their own package in `pkg/modules/migration/<name-of-the-service>`.
When creating a new migrator, you should place all related code inside that module.
### Migrator interface
The migrator interface is defined as follows:
```go
// Migrator is the basic migrator interface which is shared among all migrators
type Migrator interface {
// Migrate is the interface used to migrate a user's tasks from another platform to vikunja.
// The user object is the user who's tasks will be migrated.
Migrate(user *models.User) error
// AuthURL returns a url for clients to authenticate against.
// The use case for this are Oauth flows, where the server token should remain hidden and not
// known to the frontend.
AuthURL() string
}
```
### Defining http routes
Once your migrator implements the migration interface, it becomes possible to use the helper http handlers.
Their usage is very similar to the [general web handler](https://kolaente.dev/vikunja/web#user-content-defining-routes-using-the-standard-web-handler):
```go
// This is an example for the Wunderlist migrator
if config.MigrationWunderlistEnable.GetBool() {
wunderlistMigrationHandler := &migrationHandler.MigrationWeb{
MigrationStruct: func() migration.Migrator {
return &wunderlist.Migration{}
},
}
m.GET("/wunderlist/auth", wunderlistMigrationHandler.AuthURL)
m.POST("/wunderlist/migrate", wunderlistMigrationHandler.Migrate)
}
```
You should also document the routes with [swagger annotations]({{< ref "../practical-instructions/swagger-docs.md" >}}).
### Insertion helper method
There is a method available in the `migration` package which takes a fully nested Vikunja structure and creates it with all relations.
This means you start by adding a namespace, then add lists inside of that namespace, then tasks in the lists and so on.
The root structure must be present as `[]*models.NamespaceWithLists`.
Then call the method like so:
```go
fullVikunjaHierachie, err := convertWunderlistToVikunja(wContent)
if err != nil {
return
}
err = migration.InsertFromStructure(fullVikunjaHierachie, user)
```
### Configuration
You should add at least an option to enable or disable the migration.
Chances are, you'll need some more options for things like client ID and secret
(if the other service uses oAuth as an authentication flow).
The easiest way to implement an on/off switch is to check whether your migration service is enabled or not when
registering the routes, and then simply don't registering the routes in the case it is disabled.
#### Making the migrator public in `/info`
You should make your migrator available in the `/info` endpoint so that frontends can display options to enable them or not.
To do this, add an entry to `pkg/routes/api/v1/info.go`.

View file

@ -22,6 +22,10 @@ In general, this api repo has the following structure:
* `metrics` * `metrics`
* `migration` * `migration`
* `models` * `models`
* `modules`
* `migration`
* `handler`
* `wunderlist`
* `red` * `red`
* `routes` * `routes`
* `api/v1` * `api/v1`
@ -85,7 +89,7 @@ To learn how it works and how to add new metrics, take a look at [how metrics wo
This package handles all migrations. This package handles all migrations.
All migrations are stored and executed here. All migrations are stored and executed here.
To learn more, take a look at the [migrations docs]({{< ref "../development/migrations.md">}}). To learn more, take a look at the [migrations docs]({{< ref "../development/db-migrations.md">}}).
### models ### models
@ -97,6 +101,12 @@ Because this package is pretty huge, there are several documents and how-to's ab
* [Adding a feature]({{< ref "../practical-instructions/feature.md">}}) * [Adding a feature]({{< ref "../practical-instructions/feature.md">}})
* [Making calls to the database]({{< ref "../practical-instructions/database.md">}}) * [Making calls to the database]({{< ref "../practical-instructions/database.md">}})
### modules
#### migration
See [writing a migrator]({{< ref "migration.md" >}}).
### red (redis) ### red (redis)
This package initializes a connection to a redis server. This package initializes a connection to a redis server.

View file

@ -160,4 +160,20 @@ files:
# The maximum size of a file, as a human-readable string. # The maximum size of a file, as a human-readable string.
# Warning: The max size is limited 2^64-1 bytes due to the underlying datatype # Warning: The max size is limited 2^64-1 bytes due to the underlying datatype
maxsize: 20MB maxsize: 20MB
migration:
# These are the settings for the wunderlist migrator
wunderlist:
# Wheter to enable the wunderlist migrator or not
enable: true
# The client id, required for making requests to the wunderlist api
# You need to register your vikunja instance at https://developer.wunderlist.com/apps/new to get this
clientid:
# The client secret, also required for making requests to the wunderlist api
clientsecret:
# The url where clients are redirected after they authorized Vikunja to access their wunderlist stuff.
# This needs to match the url you entered when registering your Vikunja instance at wunderlist.
# This is usually the frontend url where the frontend then makes a request to /migration/wunderlist/migrate
# with the code obtained from the wunderlist api.
redirecturl:
{{< /highlight >}} {{< /highlight >}}

1
go.mod
View file

@ -59,7 +59,6 @@ require (
github.com/onsi/gomega v1.4.3 // indirect github.com/onsi/gomega v1.4.3 // indirect
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7 github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
github.com/pelletier/go-toml v1.4.0 // indirect github.com/pelletier/go-toml v1.4.0 // indirect
github.com/pkg/errors v0.8.1 // indirect
github.com/prometheus/client_golang v0.9.2 github.com/prometheus/client_golang v0.9.2
github.com/samedi/caldav-go v3.0.0+incompatible github.com/samedi/caldav-go v3.0.0+incompatible
github.com/shurcooL/httpfs v0.0.0-20190527155220-6a4d4a70508b github.com/shurcooL/httpfs v0.0.0-20190527155220-6a4d4a70508b

View file

@ -89,6 +89,11 @@ const (
FilesBasePath Key = `files.basepath` FilesBasePath Key = `files.basepath`
FilesMaxSize Key = `files.maxsize` FilesMaxSize Key = `files.maxsize`
MigrationWunderlistEnable Key = `migration.wunderlist.enable`
MigrationWunderlistClientID Key = `migration.wunderlist.clientid`
MigrationWunderlistClientSecret Key = `migration.wunderlist.clientsecret`
MigrationWunderlistRedirectURL Key = `migration.wunderlist.redirecturl`
) )
// GetString returns a string config value // GetString returns a string config value

View file

@ -39,6 +39,9 @@ type File struct {
CreatedByID int64 `xorm:"int(11) not null" json:"-"` CreatedByID int64 `xorm:"int(11) not null" json:"-"`
File afero.File `xorm:"-" json:"-"` File afero.File `xorm:"-" json:"-"`
// This ReadCloser is only used for migration purposes. Use with care!
// There is currentlc no better way of doing this.
FileContent []byte `xorm:"-" json:"-"`
} }
// TableName is the table name for the files table // TableName is the table name for the files table

View file

@ -0,0 +1,97 @@
// Vikunja is a todo-list application to facilitate your life.
// Copyright 2018-2020 Vikunja and contributors. All rights reserved.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <https://www.gnu.org/licenses/>.
package migration
import (
"bytes"
"code.vikunja.io/api/pkg/models"
"io/ioutil"
)
// InsertFromStructure takes a fully nested Vikunja data structure and a user and then creates everything for this user
// (Namespaces, tasks, etc. Even attachments and relations.)
func InsertFromStructure(str []*models.NamespaceWithLists, user *models.User) (err error) {
// Create all namespaces
for _, n := range str {
err = n.Create(user)
if err != nil {
return
}
// Create all lists
for _, l := range n.Lists {
// The tasks slice is going to be reset during the creation of the list so we rescue it here to be able
// to still loop over the tasks aftere the list was created.
tasks := l.Tasks
l.NamespaceID = n.ID
err = l.Create(user)
if err != nil {
return
}
// Create all tasks
for _, t := range tasks {
t.ListID = l.ID
err = t.Create(user)
if err != nil {
return
}
// Create all relation for each task
for kind, tasks := range t.RelatedTasks {
// First create the related tasks if they does not exist
for _, rt := range tasks {
if rt.ID == 0 {
err = rt.Create(user)
if err != nil {
return
}
}
// Then create the relation
taskRel := &models.TaskRelation{
TaskID: rt.ID,
OtherTaskID: t.ID,
RelationKind: kind,
}
err = taskRel.Create(user)
if err != nil {
return
}
}
}
// Create all attachments for each task
for _, a := range t.Attachments {
// Check if we have a file to create
if len(a.File.FileContent) > 0 {
a.TaskID = t.ID
fr := ioutil.NopCloser(bytes.NewReader(a.File.FileContent))
err = a.NewAttachment(fr, a.File.Name, a.File.Size, user)
if err != nil {
return
}
}
}
}
}
}
return nil
}

View file

@ -0,0 +1,66 @@
// Vikunja is a todo-list application to facilitate your life.
// Copyright 2018-2020 Vikunja and contributors. All rights reserved.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <https://www.gnu.org/licenses/>.
package handler
import (
"code.vikunja.io/api/pkg/models"
"code.vikunja.io/api/pkg/modules/migration"
"code.vikunja.io/web/handler"
"github.com/labstack/echo/v4"
"net/http"
)
// MigrationWeb holds the web migration handler
type MigrationWeb struct {
MigrationStruct func() migration.Migrator
}
// AuthURL is returned to the user when requesting the auth url
type AuthURL struct {
URL string `json:"url"`
}
// AuthURL is the web handler to get the auth url
func (mw *MigrationWeb) AuthURL(c echo.Context) error {
ms := mw.MigrationStruct()
return c.JSON(http.StatusOK, &AuthURL{URL: ms.AuthURL()})
}
// Migrate calls the migration method
func (mw *MigrationWeb) Migrate(c echo.Context) error {
ms := mw.MigrationStruct()
// Get the user from context
user, err := models.GetCurrentUser(c)
if err != nil {
return handler.HandleHTTPError(err, c)
}
// Bind user request stuff
err = c.Bind(ms)
if err != nil {
return echo.NewHTTPError(http.StatusBadRequest, "No or invalid model provided: "+err.Error())
}
// Do the migration
err = ms.Migrate(user)
if err != nil {
return handler.HandleHTTPError(err, c)
}
return c.JSON(http.StatusOK, models.Message{Message: "Everything was migrated successfully."})
}

View file

@ -0,0 +1,31 @@
// Copyright 2019 Vikunja and contriubtors. All rights reserved.
//
// This file is part of Vikunja.
//
// Vikunja is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// Vikunja is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with Vikunja. If not, see <https://www.gnu.org/licenses/>.
package migration
import "code.vikunja.io/api/pkg/models"
// Migrator is the basic migrator interface which is shared among all migrators
type Migrator interface {
// Migrate is the interface used to migrate a user's tasks from another platform to vikunja.
// The user object is the user who's tasks will be migrated.
Migrate(user *models.User) error
// AuthURL returns a url for clients to authenticate against.
// The use case for this are Oauth flows, where the server token should remain hidden and not
// known to the frontend.
AuthURL() string
}

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 KiB

View file

@ -0,0 +1,482 @@
// Vikunja is a todo-list application to facilitate your life.
// Copyright 2018-2020 Vikunja and contributors. All rights reserved.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <https://www.gnu.org/licenses/>.
package wunderlist
import (
"bytes"
"code.vikunja.io/api/pkg/config"
"code.vikunja.io/api/pkg/files"
"code.vikunja.io/api/pkg/log"
"code.vikunja.io/api/pkg/models"
"code.vikunja.io/api/pkg/modules/migration"
"code.vikunja.io/api/pkg/utils"
"encoding/json"
"fmt"
"net/http"
"net/url"
"strconv"
"time"
)
// Migration represents the implementation of the migration for wunderlist
type Migration struct {
// Code is the code used to get a user api token
Code string `query:"code" json:"code"`
}
// This represents all necessary fields for getting an api token for the wunderlist api from a code
type wunderlistAuthRequest struct {
ClientID string `json:"client_id"`
ClientSecret string `json:"client_secret"`
Code string `json:"code"`
}
type wunderlistAuthToken struct {
AccessToken string `json:"access_token"`
}
type task struct {
ID int `json:"id"`
AssigneeID int `json:"assignee_id"`
CreatedAt time.Time `json:"created_at"`
CreatedByID int `json:"created_by_id"`
DueDate string `json:"due_date"`
ListID int `json:"list_id"`
Revision int `json:"revision"`
Starred bool `json:"starred"`
Title string `json:"title"`
Completed bool `json:"completed"`
CompletedAt time.Time `json:"completed_at"`
}
type list struct {
ID int `json:"id"`
CreatedAt time.Time `json:"created_at"`
Title string `json:"title"`
ListType string `json:"list_type"`
Type string `json:"type"`
Revision int `json:"revision"`
Migrated bool `json:"-"`
}
type folder struct {
ID int `json:"id"`
Title string `json:"title"`
ListIds []int `json:"list_ids"`
CreatedAt time.Time `json:"created_at"`
CreatedByRequestID string `json:"created_by_request_id"`
UpdatedAt time.Time `json:"updated_at"`
Type string `json:"type"`
Revision int `json:"revision"`
}
type note struct {
ID int `json:"id"`
TaskID int `json:"task_id"`
Content string `json:"content"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
Revision int `json:"revision"`
}
type file struct {
ID int `json:"id"`
URL string `json:"url"`
TaskID int `json:"task_id"`
ListID int `json:"list_id"`
UserID int `json:"user_id"`
FileName string `json:"file_name"`
ContentType string `json:"content_type"`
FileSize int `json:"file_size"`
LocalCreatedAt time.Time `json:"local_created_at"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
Type string `json:"type"`
Revision int `json:"revision"`
}
type reminder struct {
ID int `json:"id"`
Date time.Time `json:"date"`
TaskID int `json:"task_id"`
Revision int `json:"revision"`
Type string `json:"type"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
type subtask struct {
ID int `json:"id"`
TaskID int `json:"task_id"`
CreatedAt time.Time `json:"created_at"`
CreatedByID int `json:"created_by_id"`
Revision int `json:"revision"`
Title string `json:"title"`
}
type wunderlistContents struct {
tasks []*task
lists []*list
folders []*folder
notes []*note
files []*file
reminders []*reminder
subtasks []*subtask
}
func convertListForFolder(listID int, list *list, content *wunderlistContents) (*models.List, error) {
l := &models.List{
Title: list.Title,
Created: list.CreatedAt.Unix(),
}
// Find all tasks belonging to this list and put them in
for _, t := range content.tasks {
if t.ListID == listID {
newTask := &models.Task{
Text: t.Title,
Created: t.CreatedAt.Unix(),
Done: t.Completed,
}
// Set Done At
if newTask.Done {
newTask.DoneAtUnix = t.CompletedAt.Unix()
}
// Parse the due date
if t.DueDate != "" {
dueDate, err := time.Parse("2006-01-02", t.DueDate)
if err != nil {
return nil, err
}
newTask.DueDateUnix = dueDate.Unix()
}
// Find related notes
for _, n := range content.notes {
if n.TaskID == t.ID {
newTask.Description = n.Content
}
}
// Attachments
for _, f := range content.files {
if f.TaskID == t.ID {
// Download the attachment and put it in the file
resp, err := http.Get(f.URL)
if err != nil {
return nil, err
}
defer resp.Body.Close()
buf := &bytes.Buffer{}
_, err = buf.ReadFrom(resp.Body)
if err != nil {
return nil, err
}
newTask.Attachments = append(newTask.Attachments, &models.TaskAttachment{
File: &files.File{
Name: f.FileName,
Mime: f.ContentType,
Size: uint64(f.FileSize),
Created: f.CreatedAt,
CreatedUnix: f.CreatedAt.Unix(),
// We directly pass the file contents here to have a way to link the attachment to the file later.
// Because we don't have an ID for our task at this point of the migration, we cannot just throw all
// attachments in a slice and do the work of downloading and properly storing them later.
FileContent: buf.Bytes(),
},
Created: f.CreatedAt.Unix(),
})
}
}
// Subtasks
for _, s := range content.subtasks {
if s.TaskID == t.ID {
if newTask.RelatedTasks[models.RelationKindSubtask] == nil {
newTask.RelatedTasks = make(models.RelatedTaskMap)
}
newTask.RelatedTasks[models.RelationKindSubtask] = append(newTask.RelatedTasks[models.RelationKindSubtask], &models.Task{
Text: s.Title,
})
}
}
// Reminders
for _, r := range content.reminders {
if r.TaskID == t.ID {
newTask.RemindersUnix = append(newTask.RemindersUnix, r.Date.Unix())
}
}
l.Tasks = append(l.Tasks, newTask)
}
}
return l, nil
}
func convertWunderlistToVikunja(content *wunderlistContents) (fullVikunjaHierachie []*models.NamespaceWithLists, err error) {
// Make a map from the list with the key being list id for easier handling
listMap := make(map[int]*list, len(content.lists))
for _, l := range content.lists {
listMap[l.ID] = l
}
// First, we look through all folders and create namespaces for them.
for _, folder := range content.folders {
namespace := &models.NamespaceWithLists{
Namespace: models.Namespace{
Name: folder.Title,
Created: folder.CreatedAt.Unix(),
Updated: folder.UpdatedAt.Unix(),
},
}
// Then find all lists for that folder
for _, listID := range folder.ListIds {
if list, exists := listMap[listID]; exists {
l, err := convertListForFolder(listID, list, content)
if err != nil {
return nil, err
}
namespace.Lists = append(namespace.Lists, l)
// And mark the list as migrated so we don't iterate over it again
list.Migrated = true
}
}
// And then finally put the namespace (which now has all the details) back in the full array.
fullVikunjaHierachie = append(fullVikunjaHierachie, namespace)
}
// At the end, loop over all lists which don't belong to a namespace and put them in a default namespace
if len(listMap) > 0 {
newNamespace := &models.NamespaceWithLists{
Namespace: models.Namespace{
Name: "Migrated from wunderlist",
},
}
for _, list := range listMap {
if list.Migrated {
continue
}
l, err := convertListForFolder(list.ID, list, content)
if err != nil {
return nil, err
}
newNamespace.Lists = append(newNamespace.Lists, l)
}
fullVikunjaHierachie = append(fullVikunjaHierachie, newNamespace)
}
return
}
func makeAuthGetRequest(token *wunderlistAuthToken, urlPart string, v interface{}, urlParams url.Values) error {
req, err := http.NewRequest(http.MethodGet, "https://a.wunderlist.com/api/v1/"+urlPart, nil)
if err != nil {
return err
}
req.Header.Set("X-Access-Token", token.AccessToken)
req.Header.Set("X-Client-ID", config.MigrationWunderlistClientID.GetString())
req.URL.RawQuery = urlParams.Encode()
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
buf := &bytes.Buffer{}
_, err = buf.ReadFrom(resp.Body)
if err != nil {
return err
}
if resp.StatusCode > 399 {
return fmt.Errorf("wunderlist API Error: Status Code: %d, Response was: %s", resp.StatusCode, buf.String())
}
// If the response is an empty json array, we need to exit here, otherwise this breaks the json parser since it
// expects a null for an empty slice
str := buf.String()
if str == "[]" {
return nil
}
return json.Unmarshal(buf.Bytes(), v)
}
// Migrate migrates a user's wunderlist lists, tasks, etc.
// @Summary Migrate all lists, tasks etc. from wunderlist
// @Description Migrates all folders, lists, tasks, notes, reminders, subtasks and files from wunderlist to vikunja.
// @tags migration
// @Accept json
// @Produce json
// @Security JWTKeyAuth
// @Param migrationCode body wunderlist.Migration true "The auth code previously obtained from the auth url. See the docs for /migration/wunderlist/auth."
// @Success 200 {object} models.Message "A message telling you everything was migrated successfully."
// @Failure 500 {object} models.Message "Internal server error"
// @Router /migration/wunderlist/migrate [post]
func (w *Migration) Migrate(user *models.User) (err error) {
log.Debugf("[Wunderlist migration] Starting wunderlist migration for user %d", user.ID)
// Struct init
wContent := &wunderlistContents{
tasks: []*task{},
lists: []*list{},
folders: []*folder{},
notes: []*note{},
files: []*file{},
reminders: []*reminder{},
subtasks: []*subtask{},
}
// 0. Get api token from oauth user token
authRequest := wunderlistAuthRequest{
ClientID: config.MigrationWunderlistClientID.GetString(),
ClientSecret: config.MigrationWunderlistClientSecret.GetString(),
Code: w.Code,
}
jsonAuth, err := json.Marshal(authRequest)
if err != nil {
return
}
resp, err := http.Post("https://www.wunderlist.com/oauth/access_token", "application/json", bytes.NewBuffer(jsonAuth))
if err != nil {
return
}
authToken := &wunderlistAuthToken{}
err = json.NewDecoder(resp.Body).Decode(authToken)
if err != nil {
return
}
log.Debugf("[Wunderlist migration] Start getting all data from wunderlist for user %d", user.ID)
// 1. Get all folders
err = makeAuthGetRequest(authToken, "folders", &wContent.folders, nil)
if err != nil {
return
}
// 2. Get all lists
err = makeAuthGetRequest(authToken, "lists", &wContent.lists, nil)
if err != nil {
return
}
for _, l := range wContent.lists {
listQueryParam := url.Values{"list_id": []string{strconv.Itoa(l.ID)}}
// 3. Get all tasks for each list
tasks := []*task{}
err = makeAuthGetRequest(authToken, "tasks", &tasks, listQueryParam)
if err != nil {
return
}
wContent.tasks = append(wContent.tasks, tasks...)
// 3. Get all done tasks for each list
doneTasks := []*task{}
err = makeAuthGetRequest(authToken, "tasks", &doneTasks, url.Values{"list_id": []string{strconv.Itoa(l.ID)}, "completed": []string{"true"}})
if err != nil {
return
}
wContent.tasks = append(wContent.tasks, doneTasks...)
// 4. Get all notes for all lists
notes := []*note{}
err = makeAuthGetRequest(authToken, "notes", &notes, listQueryParam)
if err != nil {
return
}
wContent.notes = append(wContent.notes, notes...)
// 5. Get all files for all lists
fils := []*file{}
err = makeAuthGetRequest(authToken, "files", &fils, listQueryParam)
if err != nil {
return
}
wContent.files = append(wContent.files, fils...)
// 6. Get all reminders for all lists
reminders := []*reminder{}
err = makeAuthGetRequest(authToken, "reminders", &reminders, listQueryParam)
if err != nil {
return
}
wContent.reminders = append(wContent.reminders, reminders...)
// 7. Get all subtasks for all lists
subtasks := []*subtask{}
err = makeAuthGetRequest(authToken, "subtasks", &subtasks, listQueryParam)
if err != nil {
return
}
wContent.subtasks = append(wContent.subtasks, subtasks...)
}
log.Debugf("[Wunderlist migration] Got all data from wunderlist for user %d", user.ID)
log.Debugf("[Wunderlist migration] Migrating data to vikunja format for user %d", user.ID)
// Convert + Insert everything
fullVikunjaHierachie, err := convertWunderlistToVikunja(wContent)
if err != nil {
return
}
log.Debugf("[Wunderlist migration] Done migrating data to vikunja format for user %d", user.ID)
log.Debugf("[Wunderlist migration] Insert data into db for user %d", user.ID)
err = migration.InsertFromStructure(fullVikunjaHierachie, user)
log.Debugf("[Wunderlist migration] Done inserting data into db for user %d", user.ID)
log.Debugf("[Wunderlist migration] Wunderlist migration for user %d done", user.ID)
return err
}
// AuthURL returns the url users need to authenticate against
// @Summary Get the auth url from wunderlist
// @Description Returns the auth url where the user needs to get its auth code. This code can then be used to migrate everything from wunderlist to Vikunja.
// @tags migration
// @Produce json
// @Security JWTKeyAuth
// @Success 200 {object} handler.AuthURL "The auth url."
// @Failure 500 {object} models.Message "Internal server error"
// @Router /migration/wunderlist/auth [get]
func (w *Migration) AuthURL() string {
return "https://www.wunderlist.com/oauth/authorize?client_id=" +
config.MigrationWunderlistClientID.GetString() +
"&redirect_uri=" +
config.MigrationWunderlistRedirectURL.GetString() +
"&state=" + utils.MakeRandomString(32)
}

View file

@ -0,0 +1,352 @@
// Vikunja is a todo-list application to facilitate your life.
// Copyright 2018-2020 Vikunja and contributors. All rights reserved.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <https://www.gnu.org/licenses/>.
package wunderlist
import (
"code.vikunja.io/api/pkg/config"
"code.vikunja.io/api/pkg/files"
"code.vikunja.io/api/pkg/models"
"github.com/stretchr/testify/assert"
"gopkg.in/d4l3k/messagediff.v1"
"io/ioutil"
"strconv"
"testing"
"time"
)
func TestWunderlistParsing(t *testing.T) {
config.InitConfig()
time1, err := time.Parse(time.RFC3339Nano, "2013-08-30T08:29:46.203Z")
assert.NoError(t, err)
time2, err := time.Parse(time.RFC3339Nano, "2013-08-30T08:36:13.273Z")
assert.NoError(t, err)
time3, err := time.Parse(time.RFC3339Nano, "2013-09-05T08:36:13.273Z")
assert.NoError(t, err)
time4, err := time.Parse(time.RFC3339Nano, "2013-08-02T11:58:55Z")
assert.NoError(t, err)
exampleFile, err := ioutil.ReadFile(config.ServiceRootpath.GetString() + "/pkg/modules/migration/wunderlist/testimage.jpg")
assert.NoError(t, err)
createTestTask := func(id, listID int, done bool) *task {
completedAt, err := time.Parse(time.RFC3339Nano, "1970-01-01T00:00:00Z")
assert.NoError(t, err)
if done {
completedAt = time1
}
return &task{
ID: id,
AssigneeID: 123,
CreatedAt: time1,
DueDate: "2013-09-05",
ListID: listID,
Title: "Ipsum" + strconv.Itoa(id),
Completed: done,
CompletedAt: completedAt,
}
}
createTestNote := func(id, taskID int) *note {
return &note{
ID: id,
TaskID: taskID,
Content: "Lorem Ipsum dolor sit amet",
CreatedAt: time3,
UpdatedAt: time2,
}
}
fixtures := &wunderlistContents{
folders: []*folder{
{
ID: 123,
Title: "Lorem Ipsum",
ListIds: []int{1, 2, 3, 4},
CreatedAt: time1,
UpdatedAt: time2,
},
},
lists: []*list{
{
ID: 1,
CreatedAt: time1,
Title: "Lorem1",
},
{
ID: 2,
CreatedAt: time1,
Title: "Lorem2",
},
{
ID: 3,
CreatedAt: time1,
Title: "Lorem3",
},
{
ID: 4,
CreatedAt: time1,
Title: "Lorem4",
},
{
ID: 5,
CreatedAt: time4,
Title: "List without a namespace",
},
},
tasks: []*task{
createTestTask(1, 1, false),
createTestTask(2, 1, false),
createTestTask(3, 2, true),
createTestTask(4, 2, false),
createTestTask(5, 3, false),
createTestTask(6, 3, true),
createTestTask(7, 3, true),
createTestTask(8, 3, false),
createTestTask(9, 4, true),
createTestTask(10, 4, true),
},
notes: []*note{
createTestNote(1, 1),
createTestNote(2, 2),
createTestNote(3, 3),
},
files: []*file{
{
ID: 1,
URL: "https://vikunja.io/testimage.jpg", // Using an image which we are hosting, so it'll still be up
TaskID: 1,
ListID: 1,
FileName: "file.md",
ContentType: "text/plain",
FileSize: 12345,
CreatedAt: time2,
UpdatedAt: time4,
},
{
ID: 2,
URL: "https://vikunja.io/testimage.jpg",
TaskID: 3,
ListID: 2,
FileName: "file2.md",
ContentType: "text/plain",
FileSize: 12345,
CreatedAt: time3,
UpdatedAt: time4,
},
},
reminders: []*reminder{
{
ID: 1,
Date: time4,
TaskID: 1,
CreatedAt: time4,
UpdatedAt: time4,
},
{
ID: 2,
Date: time3,
TaskID: 4,
CreatedAt: time3,
UpdatedAt: time3,
},
},
subtasks: []*subtask{
{
ID: 1,
TaskID: 2,
CreatedAt: time4,
Title: "LoremSub1",
},
{
ID: 2,
TaskID: 2,
CreatedAt: time4,
Title: "LoremSub2",
},
{
ID: 3,
TaskID: 4,
CreatedAt: time4,
Title: "LoremSub3",
},
},
}
expectedHierachie := []*models.NamespaceWithLists{
{
Namespace: models.Namespace{
Name: "Lorem Ipsum",
Created: time1.Unix(),
Updated: time2.Unix(),
},
Lists: []*models.List{
{
Created: time1.Unix(),
Title: "Lorem1",
Tasks: []*models.Task{
{
Text: "Ipsum1",
DueDateUnix: 1378339200,
Created: time1.Unix(),
Description: "Lorem Ipsum dolor sit amet",
Attachments: []*models.TaskAttachment{
{
File: &files.File{
Name: "file.md",
Mime: "text/plain",
Size: 12345,
Created: time2,
CreatedUnix: time2.Unix(),
FileContent: exampleFile,
},
Created: time2.Unix(),
},
},
RemindersUnix: []int64{time4.Unix()},
},
{
Text: "Ipsum2",
DueDateUnix: 1378339200,
Created: time1.Unix(),
Description: "Lorem Ipsum dolor sit amet",
RelatedTasks: map[models.RelationKind][]*models.Task{
models.RelationKindSubtask: {
{
Text: "LoremSub1",
},
{
Text: "LoremSub2",
},
},
},
},
},
},
{
Created: time1.Unix(),
Title: "Lorem2",
Tasks: []*models.Task{
{
Text: "Ipsum3",
Done: true,
DoneAtUnix: time1.Unix(),
DueDateUnix: 1378339200,
Created: time1.Unix(),
Description: "Lorem Ipsum dolor sit amet",
Attachments: []*models.TaskAttachment{
{
File: &files.File{
Name: "file2.md",
Mime: "text/plain",
Size: 12345,
Created: time3,
CreatedUnix: time3.Unix(),
FileContent: exampleFile,
},
Created: time3.Unix(),
},
},
},
{
Text: "Ipsum4",
DueDateUnix: 1378339200,
Created: time1.Unix(),
RemindersUnix: []int64{time3.Unix()},
RelatedTasks: map[models.RelationKind][]*models.Task{
models.RelationKindSubtask: {
{
Text: "LoremSub3",
},
},
},
},
},
},
{
Created: time1.Unix(),
Title: "Lorem3",
Tasks: []*models.Task{
{
Text: "Ipsum5",
DueDateUnix: 1378339200,
Created: time1.Unix(),
},
{
Text: "Ipsum6",
DueDateUnix: 1378339200,
Created: time1.Unix(),
Done: true,
DoneAtUnix: time1.Unix(),
},
{
Text: "Ipsum7",
DueDateUnix: 1378339200,
Created: time1.Unix(),
Done: true,
DoneAtUnix: time1.Unix(),
},
{
Text: "Ipsum8",
DueDateUnix: 1378339200,
Created: time1.Unix(),
},
},
},
{
Created: time1.Unix(),
Title: "Lorem4",
Tasks: []*models.Task{
{
Text: "Ipsum9",
DueDateUnix: 1378339200,
Created: time1.Unix(),
Done: true,
DoneAtUnix: time1.Unix(),
},
{
Text: "Ipsum10",
DueDateUnix: 1378339200,
Created: time1.Unix(),
Done: true,
DoneAtUnix: time1.Unix(),
},
},
},
},
},
{
Namespace: models.Namespace{
Name: "Migrated from wunderlist",
},
Lists: []*models.List{
{
Created: time4.Unix(),
Title: "List without a namespace",
},
},
},
}
hierachie, err := convertWunderlistToVikunja(fixtures)
assert.NoError(t, err)
assert.NotNil(t, hierachie)
if diff, equal := messagediff.PrettyDiff(hierachie, expectedHierachie); !equal {
t.Errorf("ListUser.ReadAll() = %v, want %v, diff: %v", hierachie, expectedHierachie, diff)
}
}

View file

@ -24,12 +24,13 @@ import (
) )
type vikunjaInfos struct { type vikunjaInfos struct {
Version string `json:"version"` Version string `json:"version"`
FrontendURL string `json:"frontend_url"` FrontendURL string `json:"frontend_url"`
Motd string `json:"motd"` Motd string `json:"motd"`
LinkSharingEnabled bool `json:"link_sharing_enabled"` LinkSharingEnabled bool `json:"link_sharing_enabled"`
MaxFileSize string `json:"max_file_size"` MaxFileSize string `json:"max_file_size"`
RegistrationEnabled bool `json:"registration_enabled"` RegistrationEnabled bool `json:"registration_enabled"`
AvailableMigrators []string `json:"available_migrators"`
} }
// Info is the handler to get infos about this vikunja instance // Info is the handler to get infos about this vikunja instance
@ -40,12 +41,16 @@ type vikunjaInfos struct {
// @Success 200 {object} v1.vikunjaInfos // @Success 200 {object} v1.vikunjaInfos
// @Router /info [get] // @Router /info [get]
func Info(c echo.Context) error { func Info(c echo.Context) error {
return c.JSON(http.StatusOK, vikunjaInfos{ infos := vikunjaInfos{
Version: version.Version, Version: version.Version,
FrontendURL: config.ServiceFrontendurl.GetString(), FrontendURL: config.ServiceFrontendurl.GetString(),
Motd: config.ServiceMotd.GetString(), Motd: config.ServiceMotd.GetString(),
LinkSharingEnabled: config.ServiceEnableLinkSharing.GetBool(), LinkSharingEnabled: config.ServiceEnableLinkSharing.GetBool(),
MaxFileSize: config.FilesMaxSize.GetString(), MaxFileSize: config.FilesMaxSize.GetString(),
RegistrationEnabled: config.ServiceEnableRegistration.GetBool(), RegistrationEnabled: config.ServiceEnableRegistration.GetBool(),
}) }
if config.MigrationWunderlistEnable.GetBool() {
infos.AvailableMigrators = append(infos.AvailableMigrators, "wunderlist")
}
return c.JSON(http.StatusOK, infos)
} }

View file

@ -47,6 +47,9 @@ import (
"code.vikunja.io/api/pkg/config" "code.vikunja.io/api/pkg/config"
"code.vikunja.io/api/pkg/log" "code.vikunja.io/api/pkg/log"
"code.vikunja.io/api/pkg/models" "code.vikunja.io/api/pkg/models"
"code.vikunja.io/api/pkg/modules/migration"
migrationHandler "code.vikunja.io/api/pkg/modules/migration/handler"
"code.vikunja.io/api/pkg/modules/migration/wunderlist"
apiv1 "code.vikunja.io/api/pkg/routes/api/v1" apiv1 "code.vikunja.io/api/pkg/routes/api/v1"
"code.vikunja.io/api/pkg/routes/caldav" "code.vikunja.io/api/pkg/routes/caldav"
_ "code.vikunja.io/api/pkg/swagger" // To generate swagger docs _ "code.vikunja.io/api/pkg/swagger" // To generate swagger docs
@ -372,6 +375,20 @@ func registerAPIRoutes(a *echo.Group) {
} }
a.PUT("/teams/:team/members", teamMemberHandler.CreateWeb) a.PUT("/teams/:team/members", teamMemberHandler.CreateWeb)
a.DELETE("/teams/:team/members/:user", teamMemberHandler.DeleteWeb) a.DELETE("/teams/:team/members/:user", teamMemberHandler.DeleteWeb)
// Migrations
m := a.Group("/migration")
// Wunderlist
if config.MigrationWunderlistEnable.GetBool() {
wunderlistMigrationHandler := &migrationHandler.MigrationWeb{
MigrationStruct: func() migration.Migrator {
return &wunderlist.Migration{}
},
}
m.GET("/wunderlist/auth", wunderlistMigrationHandler.AuthURL)
m.POST("/wunderlist/migrate", wunderlistMigrationHandler.Migrate)
}
} }
func registerCalDavRoutes(c *echo.Group) { func registerCalDavRoutes(c *echo.Group) {

View file

@ -1,6 +1,6 @@
// GENERATED BY THE COMMAND ABOVE; DO NOT EDIT // GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
// This file was generated by swaggo/swag at // This file was generated by swaggo/swag at
// 2019-12-07 22:54:02.661375666 +0100 CET m=+0.164990732 // 2020-01-19 16:18:04.294790395 +0100 CET m=+0.176548843
package swagger package swagger
@ -1615,6 +1615,83 @@ var doc = `{
} }
} }
}, },
"/migration/wunderlist/auth": {
"get": {
"security": [
{
"JWTKeyAuth": []
}
],
"description": "Returns the auth url where the user needs to get its auth code. This code can then be used to migrate everything from wunderlist to Vikunja.",
"produces": [
"application/json"
],
"tags": [
"migration"
],
"summary": "Get the auth url from wunderlist",
"responses": {
"200": {
"description": "The auth url.",
"schema": {
"$ref": "#/definitions/handler.AuthURL"
}
},
"500": {
"description": "Internal server error",
"schema": {
"$ref": "#/definitions/models.Message"
}
}
}
}
},
"/migration/wunderlist/migrate": {
"post": {
"security": [
{
"JWTKeyAuth": []
}
],
"description": "Migrates all folders, lists, tasks, notes, reminders, subtasks and files from wunderlist to vikunja.",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"migration"
],
"summary": "Migrate all lists, tasks etc. from wunderlist",
"parameters": [
{
"description": "The auth code previously obtained from the auth url. See the docs for /migration/wunderlist/auth.",
"name": "migrationCode",
"in": "body",
"required": true,
"schema": {
"type": "object",
"$ref": "#/definitions/wunderlist.Migration"
}
}
],
"responses": {
"200": {
"description": "A message telling you everything was migrated successfully.",
"schema": {
"$ref": "#/definitions/models.Message"
}
},
"500": {
"description": "Internal server error",
"schema": {
"$ref": "#/definitions/models.Message"
}
}
}
}
},
"/namespace/{id}": { "/namespace/{id}": {
"post": { "post": {
"security": [ "security": [
@ -4463,6 +4540,14 @@ var doc = `{
} }
} }
}, },
"handler.AuthURL": {
"type": "object",
"properties": {
"url": {
"type": "string"
}
}
},
"models.APIUserPassword": { "models.APIUserPassword": {
"type": "object", "type": "object",
"properties": { "properties": {
@ -5527,6 +5612,15 @@ var doc = `{
"type": "string" "type": "string"
} }
} }
},
"wunderlist.Migration": {
"type": "object",
"properties": {
"code": {
"description": "Code is the code used to get a user api token",
"type": "string"
}
}
} }
}, },
"securityDefinitions": { "securityDefinitions": {

View file

@ -1597,6 +1597,83 @@
} }
} }
}, },
"/migration/wunderlist/auth": {
"get": {
"security": [
{
"JWTKeyAuth": []
}
],
"description": "Returns the auth url where the user needs to get its auth code. This code can then be used to migrate everything from wunderlist to Vikunja.",
"produces": [
"application/json"
],
"tags": [
"migration"
],
"summary": "Get the auth url from wunderlist",
"responses": {
"200": {
"description": "The auth url.",
"schema": {
"$ref": "#/definitions/handler.AuthURL"
}
},
"500": {
"description": "Internal server error",
"schema": {
"$ref": "#/definitions/models.Message"
}
}
}
}
},
"/migration/wunderlist/migrate": {
"post": {
"security": [
{
"JWTKeyAuth": []
}
],
"description": "Migrates all folders, lists, tasks, notes, reminders, subtasks and files from wunderlist to vikunja.",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"tags": [
"migration"
],
"summary": "Migrate all lists, tasks etc. from wunderlist",
"parameters": [
{
"description": "The auth code previously obtained from the auth url. See the docs for /migration/wunderlist/auth.",
"name": "migrationCode",
"in": "body",
"required": true,
"schema": {
"type": "object",
"$ref": "#/definitions/wunderlist.Migration"
}
}
],
"responses": {
"200": {
"description": "A message telling you everything was migrated successfully.",
"schema": {
"$ref": "#/definitions/models.Message"
}
},
"500": {
"description": "Internal server error",
"schema": {
"$ref": "#/definitions/models.Message"
}
}
}
}
},
"/namespace/{id}": { "/namespace/{id}": {
"post": { "post": {
"security": [ "security": [
@ -4444,6 +4521,14 @@
} }
} }
}, },
"handler.AuthURL": {
"type": "object",
"properties": {
"url": {
"type": "string"
}
}
},
"models.APIUserPassword": { "models.APIUserPassword": {
"type": "object", "type": "object",
"properties": { "properties": {
@ -5508,6 +5593,15 @@
"type": "string" "type": "string"
} }
} }
},
"wunderlist.Migration": {
"type": "object",
"properties": {
"code": {
"description": "Code is the code used to get a user api token",
"type": "string"
}
}
} }
}, },
"securityDefinitions": { "securityDefinitions": {

View file

@ -13,6 +13,11 @@ definitions:
size: size:
type: integer type: integer
type: object type: object
handler.AuthURL:
properties:
url:
type: string
type: object
models.APIUserPassword: models.APIUserPassword:
properties: properties:
email: email:
@ -860,6 +865,12 @@ definitions:
version: version:
type: string type: string
type: object type: object
wunderlist.Migration:
properties:
code:
description: Code is the code used to get a user api token
type: string
type: object
info: info:
contact: contact:
email: hello@vikunja.io email: hello@vikunja.io
@ -1935,6 +1946,57 @@ paths:
summary: Login summary: Login
tags: tags:
- user - user
/migration/wunderlist/auth:
get:
description: Returns the auth url where the user needs to get its auth code.
This code can then be used to migrate everything from wunderlist to Vikunja.
produces:
- application/json
responses:
"200":
description: The auth url.
schema:
$ref: '#/definitions/handler.AuthURL'
"500":
description: Internal server error
schema:
$ref: '#/definitions/models.Message'
security:
- JWTKeyAuth: []
summary: Get the auth url from wunderlist
tags:
- migration
/migration/wunderlist/migrate:
post:
consumes:
- application/json
description: Migrates all folders, lists, tasks, notes, reminders, subtasks
and files from wunderlist to vikunja.
parameters:
- description: The auth code previously obtained from the auth url. See the
docs for /migration/wunderlist/auth.
in: body
name: migrationCode
required: true
schema:
$ref: '#/definitions/wunderlist.Migration'
type: object
produces:
- application/json
responses:
"200":
description: A message telling you everything was migrated successfully.
schema:
$ref: '#/definitions/models.Message'
"500":
description: Internal server error
schema:
$ref: '#/definitions/models.Message'
security:
- JWTKeyAuth: []
summary: Migrate all lists, tasks etc. from wunderlist
tags:
- migration
/namespace/{id}: /namespace/{id}:
post: post:
consumes: consumes: