Commit Graph

43 Commits

Author SHA1 Message Date
chrislu
645ae8c57b Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs""
This reverts commit 8cb42c39
2023-09-25 09:35:16 -07:00
chrislu
8cb42c39ad Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs"
This reverts commit 2e5aa06026, reversing
changes made to 4d414f54a2.
2023-09-18 16:12:50 -07:00
dependabot[bot]
a04bd4d26f
Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 (#4850)
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0

Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.63.1 to 1.64.0.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* API changes

* go mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2023-09-18 14:43:05 -07:00
柏杰
68feffa4e6
avoid overwriting non-nil uploadReaderToChunks.uploadErr with nil value (#4560) 2023-06-10 23:15:27 -07:00
wusong
5aec6da8a3
fix deadlock for filer upload (#4527)
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-03 13:38:27 -07:00
Chris Lu
d4566d4aaa
more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
Konstantin Lebedev
f8ef25099c
Removing chunks on failed to write to replicas (#3591)
* Removing chunks on failed to write to replicas
https://github.com/seaweedfs/seaweedfs/issues/3578

* put with in the util.Retry

* just purge on any errors
2022-09-13 22:33:35 -07:00
Konstantin Lebedev
90d55cd179
avoid data race read fileChunksSize (#3570)
https://github.com/seaweedfs/seaweedfs/issues/3557
2022-09-01 10:31:57 -07:00
Konstantin Lebedev
105702ebe0
avoid data race access to uploadReaderToChunks.uploadErr (#3550)
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
2022-08-30 00:03:01 -07:00
chrislu
3741779144 minor 2022-08-27 00:20:58 -07:00
chrislu
580a7fa685 adjust for data race
fix https://github.com/seaweedfs/seaweedfs/issues/3512
2022-08-27 00:19:13 -07:00
chrislu
fef9c6a520 filer: do not always save files in "/etc" folder to filer store
fix https://github.com/seaweedfs/seaweedfs/issues/3476
2022-08-21 17:02:16 -07:00
chrislu
28b862f45f use util.Retry to retry 2022-08-20 22:03:27 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405)
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
2022-08-04 17:35:00 -07:00
Konstantin Lebedev
22181dd018
refactor FilerRequest metrics (#3402)
* refactor FilerRequest metrics

* avoid double count proxy

* defer to
2022-08-04 01:44:54 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
banjiaojuhao
f28dbbe5c5 [bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user. 2022-03-10 11:40:39 +08:00
chrislu
bb0b784544 minor 2022-03-07 00:04:59 -08:00
banjiaojuhao
71f3046841 filer: add back isAppend function 2022-03-07 15:41:07 +08:00
banjiaojuhao
6ab09e9071 filer_http: support uploading file with offset 2022-02-22 00:15:00 +08:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Chris Lu
18c40686d9 s3: multipart upload miss data if file is chunked in 4MB
fix https://github.com/chrislusf/seaweedfs/issues/2195
2021-07-15 11:56:28 -07:00
Chris Lu
da7bd62822 /etc files are stored inside metadata store 2021-07-09 02:33:14 -07:00
Chris Lu
5e6dfbc25f locks for data racing 2021-06-07 12:04:50 -07:00
Chris Lu
452c6ef183 limits concurrent uploads for one file 2021-06-06 23:05:17 -07:00
Chris Lu
8295e2feb6 skip md5 checking for now because of race condition 2021-06-06 22:16:32 -07:00
Chris Lu
19caeb7b02 fix writing the small file 2021-06-06 20:57:03 -07:00
Chris Lu
bb45dea15a filer: parallel data upload 2021-06-06 20:23:36 -07:00
Chris Lu
44f1ba6894 refactor 2021-06-06 18:43:04 -07:00
Chris Lu
6c82326575 use bytes.Buffer to reduce memory allocation and gc 2021-06-06 13:42:36 -07:00
Chris Lu
c899bdf063 a little optimization 2021-05-06 03:03:00 -07:00
Chris Lu
3eb336e0b0 report error only for the first multipart upload
the glog.Errorf would always print for s3 multipart uploads
2021-05-06 01:57:54 -07:00
Konstantin Lebedev
c2269123d3 fix aws style Etag for chunks 2021-04-28 22:28:05 +05:00
Chris Lu
0b82edc0d2 filer: avoid stuck uploader
fix https://github.com/chrislusf/seaweedfs/issues/1980

reverting the file upload batch executor
2021-04-09 13:05:15 -07:00
Chris Lu
2327c0756b fix to avoid loop 2021-04-05 23:24:26 -07:00
Chris Lu
c131764c34 ensure error is sent back in channel
fix https://github.com/chrislusf/seaweedfs/issues/1966

avoid shared readErr variable
2021-04-05 19:40:12 -07:00
Chris Lu
8251d1140e refactor 2021-04-04 18:38:33 -07:00
Chris Lu
67e019d54b add missing changes 2021-04-02 01:11:44 -07:00
Chris Lu
7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu
cefe66f159 dedicated upload processor
avoid thundering effect of overloading volume servers
2021-04-01 02:21:40 -07:00
Chris Lu
b5880334fc refactor 2021-03-30 21:07:34 -07:00