sync

package
v0.0.0-...-c0f13c9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 26, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

README

Sync Protocol

This directory contains the Protocol Buffer definitions for the File Hub Sync Protocol, which is designed to complement the existing WebDAV implementation. The protocol is optimized for mobile devices to upload files to a home server using protobuf over HTTP/2.

Status: ✅ COMPLETE

The sync protocol implementation is complete with both HTTP REST API and gRPC support.

Features

Implemented
  • ✅ Protocol Buffer definitions (sync.proto) - 17 RPC methods
  • ✅ Generated Go code (sync.pb.go, sync_grpc.pb.go)
  • ✅ HTTP REST API (16 endpoints under /api/sync/*)
  • ✅ gRPC service with authentication interceptors
  • ✅ Chunked upload with resume capability
  • ✅ Version-based change tracking
  • ✅ Database schema and operations
  • ✅ Comprehensive unit tests
File Operations
  • GetFileInfo - Get file/directory metadata
  • ListDirectory - List directory contents with pagination
  • CreateDirectory - Create new directories
  • Delete - Delete files and directories (recursive support)
  • Move - Move/rename files and directories
  • Copy - Copy files and directories
Upload Operations
  • UploadFile - Simple upload for small files (<10MB)
  • BeginUpload - Start chunked upload session
  • UploadChunk - Upload individual chunks (1MB each)
  • FinalizeUpload - Assemble chunks and complete upload
  • CancelUpload - Cancel upload and cleanup
Download Operations
  • DownloadFile - Stream file download with conditional support
Sync Operations
  • GetCurrentVersion - Get repository version identifier
  • ListChanges - Get changes since a version (delta sync)
  • GetSyncStatus - Compare client vs server state
  • BatchOperation - Batch multiple operations (stub)

Files

  • sync.proto - Protocol Buffer definitions
  • sync.pb.go - Generated protobuf code
  • sync_grpc.pb.go - Generated gRPC service code
  • service.go - Core business logic
  • grpc_service.go - gRPC service implementation
  • grpc_auth.go - gRPC authentication interceptors
  • service_test.go - Unit tests

Building

The protobuf Go code is automatically generated during the build process:

make build

Or manually generate:

protoc --go_out=. --go_opt=paths=source_relative \
    --go-grpc_out=. --go-grpc_opt=paths=source_relative \
    pkg/sync/sync.proto

Configuration

Enable gRPC server by adding to config.yaml:

web:
  port: 8080
  grpc_port: 9090  # gRPC server port (0 to disable)

API Endpoints

HTTP REST API

All endpoints are under /api/sync/ and require authentication:

GET    /api/sync/info         - Get file info
GET    /api/sync/list         - List directory
POST   /api/sync/mkdir        - Create directory
DELETE /api/sync/delete       - Delete file/directory
POST   /api/sync/move         - Move/rename
POST   /api/sync/copy         - Copy
POST   /api/sync/upload       - Simple upload
GET    /api/sync/download     - Download file
GET    /api/sync/version      - Get current version
GET    /api/sync/changes      - Get changes since version
GET    /api/sync/status       - Get sync status
POST   /api/sync/upload/begin - Begin chunked upload
POST   /api/sync/upload/chunk - Upload chunk
POST   /api/sync/upload/finalize - Finalize upload
DELETE /api/sync/upload/cancel - Cancel upload
gRPC Service

The gRPC service runs on the configured grpc_port and provides all the same functionality with better performance for mobile clients and support for streaming downloads.

Authentication

Both HTTP and gRPC APIs support:

  • Session cookie authentication (filehub_session)
  • Basic authentication (Authorization header)
  • Token authentication (x-session-token header)

Chunked Upload Flow

  1. Call BeginUpload to create upload session
  2. Upload chunks using UploadChunk (can resume from interruption)
  3. Call FinalizeUpload to assemble and store final file
  4. Call CancelUpload to abort (optional, cleanup is automatic)

Change Tracking

The sync protocol uses version-based change tracking:

  • Each repository maintains a version string (format: v{timestamp}-{nanoseconds})
  • All operations increment the version
  • ListChanges returns changes since a specific version
  • Enables efficient delta synchronization

Documentation

Index

Constants

View Source
const (
	UserIDContextKey   contextKey = "userID"
	UsernameContextKey contextKey = "username"
	RepoNameContextKey contextKey = "repoName"
)
View Source
const (
	MaxSimpleUploadSize = 10 * 1024 * 1024 // 10MB
	ChunkSize           = 1024 * 1024      // 1MB chunks
	MaxConnectionTime   = 24 * time.Hour
	ChunkTempDir        = "chunks"
)

Variables

This section is empty.

Functions

func AuthInterceptor

func AuthInterceptor() grpc.UnaryServerInterceptor

AuthInterceptor creates a gRPC unary interceptor for authentication

func StreamAuthInterceptor

func StreamAuthInterceptor() grpc.StreamServerInterceptor

StreamAuthInterceptor creates a gRPC stream interceptor for authentication

Types

type GRPCService

type GRPCService struct {
	UnimplementedSyncServiceServer
	// contains filtered or unexported fields
}

GRPCService implements the gRPC SyncService server

func NewGRPCService

func NewGRPCService(database *bun.DB) *GRPCService

NewGRPCService creates a new gRPC service wrapper

func (*GRPCService) BatchOperation

func (g *GRPCService) BatchOperation(ctx context.Context, req *BatchOperationRequest) (*BatchOperationResponse, error)

BatchOperation implements the BatchOperation RPC

func (*GRPCService) BeginUpload

func (g *GRPCService) BeginUpload(ctx context.Context, req *BeginUploadRequest) (*BeginUploadResponse, error)

BeginUpload implements the BeginUpload RPC

func (*GRPCService) CancelUpload

func (g *GRPCService) CancelUpload(ctx context.Context, req *CancelUploadRequest) (*CancelUploadResponse, error)

CancelUpload implements the CancelUpload RPC

func (*GRPCService) Copy

func (g *GRPCService) Copy(ctx context.Context, req *CopyRequest) (*CopyResponse, error)

Copy implements the Copy RPC

func (*GRPCService) CreateDirectory

func (g *GRPCService) CreateDirectory(ctx context.Context, req *CreateDirectoryRequest) (*CreateDirectoryResponse, error)

CreateDirectory implements the CreateDirectory RPC

func (*GRPCService) Delete

func (g *GRPCService) Delete(ctx context.Context, req *DeleteRequest) (*DeleteResponse, error)

Delete implements the Delete RPC

func (*GRPCService) DownloadFile

func (g *GRPCService) DownloadFile(req *DownloadFileRequest, stream grpc.ServerStreamingServer[DownloadFileResponse]) error

DownloadFile implements the DownloadFile streaming RPC

func (*GRPCService) FinalizeUpload

func (g *GRPCService) FinalizeUpload(ctx context.Context, req *FinalizeUploadRequest) (*FinalizeUploadResponse, error)

FinalizeUpload implements the FinalizeUpload RPC

func (*GRPCService) GetCurrentVersion

func (g *GRPCService) GetCurrentVersion(ctx context.Context, req *GetCurrentVersionRequest) (*GetCurrentVersionResponse, error)

GetCurrentVersion implements the GetCurrentVersion RPC

func (*GRPCService) GetFileInfo

func (g *GRPCService) GetFileInfo(ctx context.Context, req *GetFileInfoRequest) (*GetFileInfoResponse, error)

GetFileInfo implements the GetFileInfo RPC

func (*GRPCService) GetSyncStatus

func (g *GRPCService) GetSyncStatus(ctx context.Context, req *SyncStatusRequest) (*SyncStatusResponse, error)

GetSyncStatus implements the GetSyncStatus RPC

func (*GRPCService) ListChanges

func (g *GRPCService) ListChanges(ctx context.Context, req *ListChangesRequest) (*ListChangesResponse, error)

ListChanges implements the ListChanges RPC

func (*GRPCService) ListDirectory

func (g *GRPCService) ListDirectory(ctx context.Context, req *ListDirectoryRequest) (*ListDirectoryResponse, error)

ListDirectory implements the ListDirectory RPC

func (*GRPCService) Move

func (g *GRPCService) Move(ctx context.Context, req *MoveRequest) (*MoveResponse, error)

Move implements the Move RPC

func (*GRPCService) UploadChunk

func (g *GRPCService) UploadChunk(ctx context.Context, req *UploadChunkRequest) (*UploadChunkResponse, error)

UploadChunk implements the UploadChunk RPC

func (*GRPCService) UploadFile

func (g *GRPCService) UploadFile(ctx context.Context, req *UploadFileRequest) (*UploadFileResponse, error)

UploadFile implements the UploadFile RPC

type Service

type Service struct {
	// contains filtered or unexported fields
}

func NewService

func NewService(database *bun.DB) *Service

func (*Service) BeginUpload

func (s *Service) BeginUpload(ctx context.Context, repo *model.Repository, path string, totalSize int64, userID int) (string, []int, error)

func (*Service) CancelUpload

func (s *Service) CancelUpload(ctx context.Context, uploadID string) error

func (*Service) Copy

func (s *Service) Copy(ctx context.Context, repo *model.Repository, sourcePath, destPath string, userID int) error

func (*Service) CreateDirectory

func (s *Service) CreateDirectory(ctx context.Context, repo *model.Repository, path string, userID int) error

func (*Service) Delete

func (s *Service) Delete(ctx context.Context, repo *model.Repository, path string, recursive bool, userID int) error

func (*Service) DownloadFile

func (s *Service) DownloadFile(ctx context.Context, repo *model.Repository, path string, ifNoneMatch string, userID int) (*model.FileObject, io.ReadCloser, error)

func (*Service) FinalizeUpload

func (s *Service) FinalizeUpload(ctx context.Context, uploadID string, repo *model.Repository, userID int) (string, int64, error)

func (*Service) GetCurrentVersion

func (s *Service) GetCurrentVersion(ctx context.Context, repoID int) (*model.RepositoryVersion, error)

func (*Service) GetFileInfo

func (s *Service) GetFileInfo(ctx context.Context, repo *model.Repository, path string, userID int) (*model.FileObject, error)

func (*Service) GetSyncStatus

func (s *Service) GetSyncStatus(ctx context.Context, repo *model.Repository, path string, clientETag string, clientVersion int64, userID int) (string, *model.FileObject, error)

func (*Service) ListChanges

func (s *Service) ListChanges(ctx context.Context, repoID int, sinceVersion string, maxChanges int) ([]*model.ChangeLog, error)

func (*Service) ListDirectory

func (s *Service) ListDirectory(ctx context.Context, repo *model.Repository, path string, offset, limit int, userID int) ([]*model.FileObject, int64, error)

func (*Service) Move

func (s *Service) Move(ctx context.Context, repo *model.Repository, sourcePath, destPath string, userID int) error

func (*Service) UploadChunk

func (s *Service) UploadChunk(ctx context.Context, uploadID string, chunkIndex int, data []byte) error

func (*Service) UploadFile

func (s *Service) UploadFile(ctx context.Context, repo *model.Repository, path string, data []byte, mimeType string, userID int) (string, string, int64, error)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址