mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-26 18:57:56 +08:00
Code Issues Packages Projects Releases Wiki Activity
3,005 Commits 32 Branches 296 Tags 153 MiB
d892cad15d
Commit Graph

408 Commits

Author SHA1 Message Date
Chris Lu
2b8e20f122 remove unused functions 2019-12-08 18:33:25 -08:00
Chris Lu
f38f90b7ea cleanly remove .tier volume file 2019-12-06 22:19:47 -08:00
Chris Lu
6827cabfbb volume: dataFileAccessLock change to RW Lock 2019-12-06 06:59:57 -08:00
Chris Lu
e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu
7ae8b1cc86 show volume info in volume.list in weed shell 2019-12-02 23:38:56 -08:00
Chris Lu
1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu
70648d35ad go fmt 2019-12-02 20:49:58 -08:00
Chris Lu
6383b45bd0 add lock variable 2019-12-02 20:49:50 -08:00
Chris Lu
caae543a9f fix test 2019-12-02 15:54:24 -08:00
Chris Lu
ec8de250e2 tiered storage: can copy to s3, read from s3 
master not aware tiered volume yet, file assigning is not working yet
 2019-12-02 15:08:31 -08:00
Chris Lu
0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu
61bc1d6ffd tweaking 2019-11-28 18:47:51 -08:00
Chris Lu
f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu
ee2072dff9 volume: avoid inifinite loop reading ec volume info if error happens 2019-11-27 23:07:53 -08:00
Chris Lu
0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Chris Lu
e4c63ff2f8 go fmt 2019-11-22 14:51:09 -08:00
Chris Lu
9711a6ffaa WIP 2019-11-18 19:24:37 -08:00
chenwanli
295f3beed5 rm ldb and bdb directory 2019-11-19 09:35:06 +08:00
Chris Lu
c6e8225a20 adjust memory mapped file size 
related to https://github.com/chrislusf/seaweedfs/pull/1087
 2019-11-12 21:05:48 -08:00
Lei Liu
46755ea1e1 fix master maintenance logic 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-12 14:46:10 +08:00
Chris Lu
85f8649320 refactor memory mapped file into backend storage 2019-11-09 00:10:59 -08:00
Chris Lu
c5c1d83d91 Merge branch 'master' into refactoring_dat_backend 2019-11-08 22:54:41 -08:00
Chris Lu
c34ffed43f go fmt 2019-11-08 22:47:50 -08:00
Chris Lu
84c503c6a7 adjust ec reading log level 2019-11-08 22:41:02 -08:00
zhangsong
1dd101f782 use read lock to avoid io hang during heartbeat 2019-11-08 20:27:55 +08:00
Chris Lu
db30a46050 skip memory mapped file when compacting 2019-10-29 23:34:38 -07:00
Chris Lu
5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Chris Lu
57e441d67b fix compaction logic 2019-10-29 23:18:01 -07:00
Chris Lu
b7156291a8 adjust variable names 2019-10-29 23:16:43 -07:00
Chris Lu
4b5ba4927f adjust NewDiskFile() paramaters 2019-10-29 22:37:36 -07:00
Chris Lu
cd7ce720c7 fix test 2019-10-29 00:37:15 -07:00
Chris Lu
19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu
7a51a9a582 refactor: simplifying to ReadAt() and WriteAt() 2019-10-25 00:11:25 -07:00
Chris Lu
46ed2ca902 refactoring 2019-10-24 23:41:32 -07:00
Chris Lu
f937933b3d refactoring 2019-10-23 22:25:53 -07:00
Chris Lu
3c865ee39b fix missing needle header read 2019-10-22 00:57:47 -07:00
Chris Lu
fc412e428b refactor ScanVolumeFileFrom() 2019-10-22 00:50:30 -07:00
Chris Lu
c9a183eb69 refactor memory map related code 2019-10-22 00:49:42 -07:00
Chris Lu
fec07c829d go fmt 2019-10-21 23:03:48 -07:00
Chris Lu
faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
j.laycock
2e2fe00dbd Comma, no space? 2019-10-18 11:31:25 +01:00
j.laycock
248f3be6e3 using a space instead of a comma to hopefully fix the build! 2019-10-18 11:23:02 +01:00
j.laycock
2c455841ea Make volumeCreate more consistent between all 3 implementations. 2019-10-18 11:01:45 +01:00
j.laycock
b0ddad6889 Fix volume_create breaking the build 2019-10-18 10:32:07 +01:00
joeslay
d53aee179b
Merge pull request #10 from chrislusf/master 
merge seaweed master
 2019-10-14 16:03:40 +01:00
Chris Lu
87e5a02a99 fix logic error 2019-10-09 00:07:18 -07:00
Chris Lu
09874f0d16 volume: return error if superblock is not initialized 
fix https://github.com/chrislusf/seaweedfs/issues/1079
 2019-10-09 00:02:50 -07:00
j.laycock
eb27c2b037 Make releaseMemory private and return byte array instead, fix other platform compilation issues, reduce in-memory chunk size. 2019-10-01 12:21:44 +01:00
j.laycock
d5f5acb734 limit locking physical memory to 80% of max physical memory 2019-09-20 12:44:29 +01:00
j.laycock
476140fd6b minor change to setProcessWorkingSetSize function 2019-09-19 11:59:00 +01:00
j.laycock
6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
joeslay
d8c34b032f
Merge pull request #6 from chrislusf/master 
merge seaweed master
 2019-09-12 11:44:09 +01:00
Chris Lu
5e9c65469e volume: skip readonly checking when Destroy() 
fix https://github.com/chrislusf/seaweedfs/issues/1063
 2019-09-11 09:45:09 -07:00
j.laycock
c1a928887a Add some comments/documentation 2019-09-11 10:42:37 +01:00
joeslay
9ea9a5e97a
Merge pull request #5 from chrislusf/master 
merge seaweed master
 2019-09-11 10:25:27 +01:00
j.laycock
15a4c91c28 Remove Finalizer 2019-09-10 17:05:54 +01:00
j.laycock
9c9dff7386 Virtual lock memory pages and Set Process Working set size to hint to windows harder not to write pages to disk as much, add finalize function, minor renaming 2019-09-10 16:18:51 +01:00
Chris Lu
02710f9624 refactoring: remove unused functions 2019-09-09 10:03:41 -07:00
Chris Lu
0f861d23a2 avoid nil needle map 
fix https://github.com/chrislusf/seaweedfs/issues/1061
 2019-09-09 09:48:08 -07:00
j.laycock
63acc8c972 rename mem_buffer to mBuffer 2019-09-05 13:46:29 +01:00
j.laycock
44ae041e80 rename more memorymapped variables to memoryMapMaxSizeMB 2019-09-05 13:42:23 +01:00
j.laycock
5885ab67b3 rename volume property MemoryMap to MemoryMapMaxSizeMB 2019-09-05 11:56:48 +01:00
j.laycock
5a6dfb05fc Only use truncate flag when creating non memory map files, rename and use memory map size for reserving max file size 2019-09-05 11:46:12 +01:00
j.laycock
f6146d6ad2 use 0 instead of false for max memory flag 2019-09-05 11:43:18 +01:00
j.laycock
4257582db5 Allocate in 16MB chunks, make creation of memory maps always aligned to 16MB chunks 2019-09-05 11:15:01 +01:00
Tom Maxwell
4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock
cc756ddd79 Rename End_of_file variable, implement platform indepenent memory_map.go 2019-09-04 14:24:17 +01:00
j.laycock
de5a7e60a2 remove temp hack 2019-09-03 18:20:09 +01:00
j.laycock
cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
j.laycock
1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
j.laycock
9a459d984b Do not vacuum memory mapped files 2019-09-03 15:05:43 +01:00
j.laycock
0123c7a898 Change to max 2GB for now 2019-09-02 17:46:22 +01:00
joeslay
e91139348e
Merge pull request #2 from chrislusf/master 
merge seaweed master
 2019-09-02 11:39:40 +01:00
j.laycock
595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
j.laycock
1e62a2b233 Fix memory not being unmapped 2019-08-30 16:55:33 +01:00
j.laycock
840ccdc35d Refactor to pass memory maps by reference instead of value, fix memory maps not being created properly or written to properly 2019-08-30 12:30:00 +01:00
j.laycock
dc50701e7d Overload createFile function, expose File in memory_map struct, delete from memory_map map on delete 2019-08-30 12:30:00 +01:00
j.laycock
523f3a12b3 use Os.File/uintptr instead of windows.Handle in memory map struct 2019-08-30 12:30:00 +01:00
j.laycock
0e5d3b1a70 Move memory_map_windows to new folder, intercept most of the read and write calls to volume dat files 2019-08-30 12:30:00 +01:00
Chris Lu
58168a8c52 volume: truncate dat files when creating 2019-08-26 13:01:57 -07:00
joeslay
36bde84000
Merge pull request #1 from chrislusf/master 
merge changes from chrislusf/seaweed master
 2019-08-23 17:03:37 +01:00
j.laycock
1e89e719b3 Add windows memory map functions to storage package 2019-08-23 16:19:00 +01:00
xushuxun
9af9f36b89 volume: read dat files' last modified time no matter dat files are catWrite or readonly 2019-08-21 11:26:01 +08:00
Chris Lu
f9c1cd7370 avoid dead lock from dataFileAccessLock 
fix https://github.com/chrislusf/seaweedfs/issues/1043
 2019-08-15 09:24:54 -07:00
Chris Lu
d829df4f59 volume: protect against nil needle map 
fix @mastak reported nil problem in https://github.com/chrislusf/seaweedfs/issues/1037
 2019-08-14 01:08:01 -07:00
Chris Lu
e40634e6b4 volume: fail the volume deletion if compaction is in progress 
fix https://github.com/chrislusf/seaweedfs/issues/1035
 2019-08-12 00:53:50 -07:00
Chris Lu
f3b99cbfe5 fix wrong alignment fix 2019-07-21 13:50:42 -07:00
Chris Lu
644c5c27b4 refactoring 2019-07-21 13:50:24 -07:00
Chris Lu
33b3366043 volume: avoid compacting wait group 2019-07-21 13:49:59 -07:00
Chris Lu
fd16adbde1 stricter checking before writes 2019-07-21 11:21:30 -07:00
Chris Lu
5a135fef87 purge old code 2019-07-21 10:45:36 -07:00
Chris Lu
c54d9221b9 write requests also checks cookie if overwrites 
protect against edge cases, avoid https://github.com/chrislusf/seaweedfs/issues/1014
 2019-07-17 23:57:34 -07:00
Chris Lu
898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu
c33f423955 stop early if compaction fails 
fix https://github.com/chrislusf/seaweedfs/issues/1015
 2019-07-17 23:22:01 -07:00
Chris Lu
a7b1b23c58 fix wrong volume count 
fix https://github.com/chrislusf/seaweedfs/issues/1013
 2019-07-17 00:03:15 -07:00
Chris Lu
a3d1296ed9 go fmt 2019-06-27 12:18:59 -07:00
Chris Lu
ec75b2d761 volume: fix bug with 8000GB version if using in memory index 
fix https://github.com/chrislusf/seaweedfs/issues/994
 2019-06-27 12:18:45 -07:00
Chris Lu
6883f9e322 mark volume readonly before ec encoding 2019-06-26 23:02:22 -07:00
Chris Lu
c2960c7a86 defaulting file ttl to volume ttl 
fix https://github.com/chrislusf/seaweedfs/issues/992
 2019-06-26 11:18:23 -07:00