mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-21 06:57:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,838 Commits 32 Branches 296 Tags 152 MiB
d440135d70
Commit Graph

377 Commits

Author SHA1 Message Date
Chris Lu
7f90d14f10 Java: add SeaweedOutputStream example 2021-02-04 20:16:08 -08:00
Chris Lu
502554887f Java: add SeaweedInputStream example 2021-02-04 18:44:57 -08:00
Chris Lu
87d1bfa862 Hadoop Compatible FS: 1.5.7 2021-01-30 06:18:41 -08:00
Chris Lu
6f4aab51f9 refactoring SeaweedInputStream 2021-01-30 06:16:02 -08:00
Chris Lu
043c2d7960 refactoring SeaweedOutputStream 2021-01-30 05:39:09 -08:00
Chris Lu
94525aa0fd allocate volume by disk type 2020-12-13 23:08:21 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu
4d2855476c Hadoop: add BufferedByteBufferReadableInputStream 
fix https://github.com/chrislusf/seaweedfs/issues/1645
 2020-12-03 00:08:05 -08:00
Chris Lu
3857f9c840 Hadoop: switch to ByteBuffer 
fix https://github.com/chrislusf/seaweedfs/issues/1645
 2020-12-02 23:45:39 -08:00
Chris Lu
a9efaa6385 HDFS: implement ByteBufferReadable 
fix https://github.com/chrislusf/seaweedfs/issues/1645
 2020-12-01 17:20:31 -08:00
Chris Lu
f4abd01adf filer: cache small file to filer store 2020-11-30 04:34:04 -08:00
Thilo-Alexander Ginkel
ec51d77dcf
grafana: remove incorrect QPS factor 2020-11-23 12:00:36 +01:00
Chris Lu
da04bb3d1b 1.5.6 2020-11-18 01:46:38 -08:00
limd
4737df597d HCFS: 
1. add replication parameter
2. fix close sequence
 2020-11-17 19:05:57 +08:00
hilimd
aec071187e
Merge pull request #38 from chrislusf/master 
sync
 2020-11-17 19:01:04 +08:00
Chris Lu
dc304342b2 fs.configure: configurable volume growth 2020-11-17 01:00:02 -08:00
Chris Lu
5f19e81dab filer confi: support hierachical configuration 2020-11-16 16:50:12 -08:00
hilimd
218561c206
Merge pull request #37 from chrislusf/master 
sync
 2020-11-16 16:30:19 +08:00
Chris Lu
fe01059675 volume id cache: reduce cache time 2020-11-15 21:27:21 -08:00
Chris Lu
61d96fde01 protect against edge cases when locations expires 2020-11-15 21:26:04 -08:00
Chris Lu
95c0de285d refactoring 2020-11-15 16:58:48 -08:00
Chris Lu
824e96ffcc updates 2020-11-13 12:47:46 -08:00
Chris Lu
3e362451d2 add example of watch files 2020-11-13 12:10:55 -08:00
Chris Lu
0d5355c614 rename 2020-11-13 11:43:25 -08:00
Chris Lu
7d24980840 HCFS: 1.5.4 2020-11-13 11:37:53 -08:00
hilimd
e0d5207ed9
Merge pull request #36 from chrislusf/master 
sync
 2020-11-13 15:34:38 +08:00
Chris Lu
a2962604ad add volume id cache 2020-11-12 23:25:11 -08:00
Chris Lu
559dfacdac adding test example 2020-11-12 23:24:39 -08:00
Chris Lu
a4dfa00030 adjust filer conf 2020-11-12 00:39:59 -08:00
Chris Lu
c32b6f4d9d prepare for path based configuration 2020-11-11 23:46:45 -08:00
hilimd
a38efe2e7b
Merge pull request #35 from chrislusf/master 
sync
 2020-11-11 10:16:39 +08:00
Chris Lu
4f676aa7d3 2.09 2020-11-08 22:06:06 -08:00
hilimd
546f1bcb90
Merge pull request #34 from chrislusf/master 
sync
 2020-11-05 12:02:47 +08:00
Chris Lu
c709059b69 HCFS: add close() to SeaweedFileSystem.java 2020-11-03 00:46:00 -08:00
hilimd
843865f2ca
Merge pull request #33 from chrislusf/master 
sync
 2020-10-26 22:01:50 +08:00
Chris Lu
f375b93aef renaming 2020-10-25 15:32:43 -07:00
Chris Lu
596d476e3d HCFS: 1.5.2 2020-10-25 14:44:18 -07:00
hilimd
ab1105c524
Merge pull request #31 from chrislusf/master 
sync
 2020-10-21 20:04:11 +08:00
Chris Lu
6c92fa349d filer: support default rack 
fix https://github.com/chrislusf/seaweedfs/issues/1546
 2020-10-20 17:41:39 -07:00
hilimd
6e63e5eaf0
Merge pull request #29 from chrislusf/master 
sync
 2020-10-19 09:42:43 +08:00
Chris Lu
b7c224cf61 HCFS: 1.5.1 2020-10-18 14:30:01 -07:00
Chris Lu
459de70a77 Hadoop: more accurate block size 2020-10-18 14:13:46 -07:00
hilimd
62af2d961d
Merge pull request #28 from chrislusf/master 
sync
 2020-10-18 09:44:04 +08:00
Chris Lu
ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Konstantin Lebedev
6f501a9f0e add service monitors 2020-10-15 18:10:22 +05:00
Konstantin Lebedev
d59446cbb7 add grafana seaweedfs dashboard for k8s 2020-10-15 17:35:22 +05:00
hilimd
a911375798
Merge pull request #26 from chrislusf/master 
sync
 2020-10-14 16:39:45 +08:00
Chris Lu
cad7fd8775
Merge pull request #1524 from chrislusf/dependabot/maven/other/java/s3copier/junit-junit-4.13.1 
Bump junit from 3.8.1 to 4.13.1 in /other/java/s3copier
 2020-10-12 11:46:07 -07:00
Chris Lu
4e31ca9507 upgrade junit to 4.13.1 2020-10-12 11:37:15 -07:00
dependabot[bot]
b57d435abe
Bump junit from 3.8.1 to 4.13.1 in /other/java/s3copier 
Bumps [junit](https://github.com/junit-team/junit4) from 3.8.1 to 4.13.1.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.1.md)
- [Commits](https://github.com/junit-team/junit4/commits/r4.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-10-12 18:14:12 +00:00
Chris Lu
253323ce6d Hadoop 1.5.0 2020-10-12 11:13:38 -07:00
hilimd
2ff727a32d
Merge pull request #25 from chrislusf/master 
sync
 2020-10-12 11:01:13 +08:00
Chris Lu
d5fcb0f474 Hadoop: printout the entry creation error message 2020-10-11 15:11:16 -07:00
Chris Lu
ea13e0523b fix logging 2020-10-11 14:59:48 -07:00
Chris Lu
8e2f3b510e Hadoop client: 1.4.9 2020-10-10 15:43:04 -07:00
hilimd
b9a446839a
Merge pull request #24 from chrislusf/master 
sync
 2020-10-10 16:20:26 +08:00
Chris Lu
912ef2bc53 Hadoop: remove unused variable bufferSize 2020-10-10 01:14:17 -07:00
Chris Lu
9449100757 adjust throwing exception 2020-10-10 01:06:54 -07:00
Chris Lu
d3d3609a03 adjust logging 2020-10-10 01:04:31 -07:00
Chris Lu
6fc272f913 adjust throwing exception 2020-10-10 01:03:04 -07:00
Chris Lu
00a75d7c99 Hadoop: fix reading file tail 2020-10-10 00:57:57 -07:00
Chris Lu
e1a3ffcdbf Hadoop: add exponential back off for failed reads 2020-10-10 00:57:57 -07:00
hilimd
411e49f964
Merge pull request #23 from chrislusf/master 
sync
 2020-10-10 13:07:57 +08:00
limd
ac162fc857 hdfs: Hadoop on SeaweedFS: create empty file 2020-10-10 13:07:27 +08:00
Chris Lu
a34bad2cee moving grafana dashboard here 2020-09-30 13:30:21 -07:00
Chris Lu
436ab7d811 HCFS: 1.4.8 2020-09-27 23:20:08 -07:00
Chris Lu
4929d0634e Hadoop on SeaweedFS: create empty file 
fix https://github.com/chrislusf/seaweedfs/issues/1494
 2020-09-25 10:57:43 -07:00
limd
95bfec4931 hadoop: filesystem cannot create file 
issues: https://github.com/chrislusf/seaweedfs/issues/1494
 2020-09-26 01:38:48 +08:00
Chris Lu
1012df7bb5 switch hardlink id from int64 to bytes 2020-09-24 11:11:42 -07:00
Chris Lu
5e239afdfc hardlink works now 2020-09-24 03:06:48 -07:00
Chris Lu
cb427d48fa filer report metrics configuration 2020-09-17 06:46:51 -07:00
Chris Lu
5eee4983f3 1.4.7 hdfs configurable fs.seaweed.buffer.size 2020-09-16 17:18:18 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
bba90ff3c8 read filer signature 2020-09-05 22:52:15 -07:00
Chris Lu
ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
Chris Lu
48ed7f8a32 generated files 2020-08-16 21:25:24 -07:00
Chris Lu
f5d6b8b777 Hadoop HCFS 1.4.6 random access file 2020-08-16 21:17:03 -07:00
Chris Lu
13bfe5deef same logic for reading random access files from Go 2020-08-16 21:14:39 -07:00
Chris Lu
20e0bae5d1 add for testing 2020-08-16 09:03:18 -07:00
Chris Lu
24bfd26719 make it easy to test randome access file 2020-08-15 14:37:07 -07:00
Chris Lu
4f195a54ca 1.4.5 
change the scope for hadoop depencies to provided
 2020-08-09 16:34:47 -07:00
Chris Lu
15dc0a704d Revert "add read ahead input stream" 
This reverts commit b3089dcc8e.
 2020-08-03 09:11:24 -07:00
Chris Lu
b3089dcc8e add read ahead input stream 2020-08-03 09:06:09 -07:00
Chris Lu
53190a9972 fix compilation 2020-08-03 09:05:49 -07:00
Chris Lu
d6073f6386 ensure GC 2020-08-03 00:40:23 -07:00
Chris Lu
9981748498 only try to cache chunk manifest chunks 2020-08-02 23:50:44 -07:00
Chris Lu
0ed1f43d29 decompress after decrypt if necessary 
skip any decompress error
 2020-08-02 23:50:23 -07:00
Chris Lu
e1f070a9a1 Hadoop: 1.4.4 2020-08-01 12:42:41 -07:00
Chris Lu
4d9da157bc HDFS: read gzip content 2020-08-01 11:36:29 -07:00
Chris Lu
8cc35e2c13 fix wrong decompression logic 2020-08-01 11:19:57 -07:00
Chris Lu
6b41c5250b Hadoop file system: 1.4.3 
added buffered fs input stream
 2020-07-29 21:33:10 -07:00
Chris Lu
703057bff9 mirror changes from hdfs2 2020-07-29 20:15:43 -07:00
Chris Lu
a6d0f962e8 add buffered seaweed input stream 2020-07-29 20:10:04 -07:00
Chris Lu
5080bc1d69 debug 2020-07-29 18:26:26 -07:00
Chris Lu
daf8c0c8ce cache vid locations 2020-07-29 18:25:52 -07:00
Chris Lu
b684c312d2 minor 2020-07-29 18:25:35 -07:00
Chris Lu
735912e0f5 enable read cache 2020-07-29 17:39:44 -07:00
Chris Lu
6f058b3791 HCFS: 1.4.1 2020-07-22 22:56:53 -07:00
Chris Lu
648ef566c4 HCFS: avoid lock bottleneck 2020-07-22 22:54:06 -07:00
Chris Lu
6839f96c0c simplify 2020-07-22 22:52:49 -07:00
Chris Lu
54f9f9138e HCFS: 1.4.0 2020-07-20 23:25:08 -07:00
Chris Lu
64926d2345 disable ChunkCache 2020-07-20 21:18:58 -07:00
Chris Lu
ae3e6d8244 remove changing buffer size 2020-07-20 20:27:19 -07:00
Chris Lu
b380b9e5d7 shared http client 2020-07-20 18:27:11 -07:00
Chris Lu
0e341a2a9a error logging 2020-07-20 18:26:59 -07:00
Chris Lu
cacc601cc8 ensure changing buffer size requirements 2020-07-20 18:26:48 -07:00
Chris Lu
aee6d89350 before writing and after reading file chunks 2020-07-20 18:26:17 -07:00
Chris Lu
6debe3c3ec use 1000 as merge factor 2020-07-20 18:25:26 -07:00
Chris Lu
1d724ab237 hdfs: support read write chunk manifest 2020-07-20 03:29:17 -07:00
Chris Lu
f90d2c93c9 1.3.9 remove logs 2020-07-17 16:11:57 -07:00
Chris Lu
3abd74b1d7 1.3.8 2020-07-17 15:46:20 -07:00
Chris Lu
0b9d73fb7c async copy 2020-07-17 15:46:20 -07:00
Chris Lu
54445b207d Hadoop: 1.3.6 2020-07-16 16:20:01 -07:00
Chris Lu
d6d95765f9 reduce chunk cache size 2020-07-16 16:19:01 -07:00
Chris Lu
2629da2cb9 simplify inputstream 2020-07-16 10:54:29 -07:00
Chris Lu
c9b50e8a22 copy changes to hadoop2 2020-07-16 09:05:34 -07:00
李明达
74456b3d5e 1、Add SeaweedFS implementation of Hadoop AbstractFileSystem. The implementation delegates to the existing SeaweedFS FileSystem and is only necessary for use with Hadoop 2.x/3.x. Configuration example in Hadoop core-site.xml file: 
<property>
    <name>fs.AbstractFileSystem.seaweedfs.impl</name>
    <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
 </property>

2、Fix hiveserver2 startup NullPointerException
 2020-07-16 16:47:09 +08:00
Chris Lu
64d7d5fe89 increase inbound message size 2020-07-16 00:33:54 -07:00
Chris Lu
7bca72deed reuse bytebuffer 2020-07-15 23:33:31 -07:00
Chris Lu
2286d27730 rename 2020-07-15 14:19:15 -07:00
Chris Lu
86c8137546 1.3.4 2020-07-15 13:49:06 -07:00
Chris Lu
22f8aae028 fix compilation 2020-07-15 13:27:10 -07:00
Chris Lu
bc3be0bb37 Hadoop: 1.3.3 
improve memory efficiency
 2020-07-15 13:25:44 -07:00
Chris Lu
316e853e0e option to disable chunk cache 2020-07-15 09:51:51 -07:00
Chris Lu
0db4204c81 add is_file_chunks to filer.proto 2020-07-14 22:42:52 -07:00
Chris Lu
fc9af962ec 1.3.2 2020-07-12 14:51:24 -07:00
Chris Lu
4bd8f3281e 1.3.0 
* add option to include last entry
*
 2020-07-11 22:24:02 -07:00
Chris Lu
578f316173 refactoring 2020-07-11 15:33:31 -07:00
Chris Lu
70d8a3a1d3 add SubscribeLocalMetadata without checking persisted meta logs 2020-07-05 15:50:07 -07:00
Chris Lu
37d5b3ba12 replication: pass isFromOtherCluster also to EventNotification 
EventNotification is consistent with message queue and metadata logs.
 2020-07-01 08:06:20 -07:00
Chris Lu
31e23e9783 filer: support active<=>active filer replication 2020-06-30 22:53:57 -07:00
Chris Lu
97239ce6f1 rename filechunk is_gzipped to is_compressed 2020-06-20 08:15:49 -07:00
Chris Lu
ca3516ac6d adjust protoc 2020-06-20 08:00:25 -07:00
Chris Lu
ce79ec0bd4 HCFS: 1.2.9 fix concurrent modification 2020-06-18 07:52:32 -07:00
Chris Lu
222f93e816 possibly fix concurrent access to entry object in SeaweedOutputStream 2020-06-17 14:59:43 -07:00
Chris Lu
8dfaaeabfd HCFS: 1.2.8 fix hbase related bugs 
1. SeaweedFileSystem.listStatus need to work with file also
2. SeaweedRead readChunkView has wrong len
 2020-05-24 17:07:34 -07:00
Chris Lu
dfccc3c263 able to read chan and write chan 2020-05-08 02:47:22 -07:00
Chris Lu
1e3e4b3072 add broker connects to filer 2020-05-05 02:05:28 -07:00
Chris Lu
972f437225 HCFS 1.2.7 2020-04-26 05:44:56 -07:00
Chris Lu
b52b8ec685 Hadoop: fix entry not found for HCFS 
also fix cipher related changes.
 2020-04-26 05:21:54 -07:00
Chris Lu
e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu
bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu
ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu
2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu
bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu
50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu
5ebf232e79 HDFS: 1.2.6 2020-03-29 01:40:04 -07:00
Chris Lu
b656e05aaf HDFS: support chunk cache 2020-03-29 00:55:40 -07:00
Chris Lu
ae2309dc58 Update SeaweedRead.java 2020-03-28 23:58:59 -07:00
Chris Lu
3d3bab2447 add notes 2020-03-28 23:43:24 -07:00
Chris Lu
c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu
de1ba85346 HDFS support encrypted data storage 2020-03-14 00:27:57 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu
0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu
bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu
6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu
d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
6a5c037099 fix http range requests 2020-01-22 22:59:12 -08:00
Chris Lu
d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu
69ffcbdcc0 HCFS: release 1.2.4 2019-12-14 11:04:36 -08:00
Chris Lu
987108a2b1 HCFS: streaming list files 2019-12-13 10:04:53 -08:00
Chris Lu
0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu
e9dd5a602a HCFS: 1.2.3 
shade org.apache.http in https://github.com/chrislusf/seaweedfs/issues/1080
 2019-10-11 23:56:04 -07:00
Chris Lu
7057e7a076 HCFS: 1.22 
fix put gives java.lang.NoSuchFieldError: INSTANCE related to Cloudera CDH Hadoop https://github.com/chrislusf/seaweedfs/issues/1080
 2019-10-11 21:44:44 -07:00
Chris Lu
15f968b054 HCFS: avoid possible jar problem 2019-10-09 18:09:08 -07:00
Chris Lu
a999ed94d0 update hdfs client 2019-10-08 22:49:10 -07:00
Chris Lu
ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Chris Lu
d361ecb23b HCFS: 1.1.6 2019-09-03 01:31:31 -07:00
Chris Lu
fd509c3844 HCFS: working with HBase 2019-09-03 00:50:28 -07:00
Chris Lu
60c9215a00 HCFS: avoid verbose file not found exception stack trace 2019-09-01 21:40:26 -07:00
Chris Lu
af8b413a9c HCFS: fix mkdirs NPE 2019-09-01 12:25:06 -07:00
Chris Lu
cb299dfaa2 HCFS: use latest grpc versions, separate hadoop2 and hadoop3 2019-09-01 03:46:51 -07:00
Chris Lu
170ee6ef0f tmp 2019-08-29 23:29:10 -07:00
Chris Lu
8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu
82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu
b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu
97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu
097b7a321e HDFS: move to 1.0.8 2019-02-20 00:57:31 -08:00
Chris Lu
6ed69de6bd HDFS: add jwt auth for uploading 2019-02-19 14:26:59 -08:00
Chris Lu
58d4088db4 HDFS: add tls secured grpc 2019-02-19 11:57:25 -08:00
Chris Lu
74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu
9123d799b8 refactor 2018-12-27 23:40:23 -08:00
Chris Lu
319ab6d98f refactoring 2018-12-27 23:29:51 -08:00
Chris Lu
253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu
9546104961 HCFS 1.05 2018-12-18 20:23:40 -08:00
Chris Lu
2c72447078 HCFS: bug fix for listEntries 
fix https://github.com/chrislusf/seaweedfs/issues/797
 2018-12-18 20:22:51 -08:00
Chris Lu
e716ec9908 HCFS 1.04 2018-12-17 20:40:10 -08:00
Chris Lu
cab7fc7738 fix directory creation 
fix https://github.com/chrislusf/seaweedfs/issues/794
 2018-12-17 20:37:55 -08:00
Chris Lu
d11cfdb356 paginate through large folders 2018-12-17 01:25:02 -08:00
Chris Lu
21315f709d HCFS 1.0.2 2018-12-14 09:16:21 -08:00
Chris Lu
36d1e739e6 1.11 2018-12-08 00:32:34 -08:00
Chris Lu
adf12c8825 refactoring 2018-12-07 02:23:45 -08:00
Chris Lu
1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu
6946c51430 avoid range request for gzipped content 2018-12-07 01:23:30 -08:00