I received the following error:
FailedPreconditionError: /home/timucin/Thesis/Tagger-master8/train; Is a directory
[[{{node input_queues/examples_queue/parallel_read/ReaderReadV2}} = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](input_queues/examples_queue/parallel_read/TFRecordReaderV2, input_queues/examples_queue/parallel_read/filenames)]]
While running a training session of semantic role labeling. I`m using python 2.7 (anaconda) with TensorFlow 1.12 on Ubuntu 18.04.
When I first run the script, train directory is created to store checkpoints and the session terminates with no error.
During the second trial, with the train directory already created, the script looks at the checkpoints to assess whether there are new steps on top of already completed ones and terminates with the above mentioned error. When I delete the content of the directory I receive the same error. But if I delete the directory error disappears.
I guess the file I/O is the reason that`s why I post it to this forum.
I would appreciate to receive your feedback on this... Thanks in advance, Timucin
Trace back:
INFO:tensorflow:Total trainable variables size: 8187305
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from train/model.ckpt-10
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.FailedPreconditionError'>, /home/timucin/Thesis/Tagger-master8/train; Is a directory
[[{{node input_queues/examples_queue/parallel_read/ReaderReadV2}} = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](input_queues/examples_queue/parallel_read/TFRecordReaderV2, input_queues/examples_queue/parallel_read/filenames)]]
INFO:tensorflow:Saving checkpoints for 10 into train/model.ckpt.
Traceback (most recent call last):
File "<ipython-input-6-5b359d738d7d>", line 1, in <module>
runfile('/home/timucin/Thesis/Tagger-master8/main.py', args='train --model_name deepatt --vocab_path /home/timucin/Thesis/Tagger-master-old/word_dict /home/timucin/Thesis/Tagger-master-old/label_dict --data_path /home/timucin/Thesis/Tagger-master8/ --model_dir train --model_params=feature_size=100,hidden_size=200,filter_size=800,residual_dropout=0.2,num_hidden_layers=10,attention_dropout=0.1,relu_dropout=0.1 --training_params=batch_size=4096,eval_batch_size=1024,optimizer=Adadelta,initializer=orthogonal,use_global_initializer=false,initializer_gain=1.0,train_steps=15,learning_rate_decay=piecewise_constant,learning_rate_values=[1.0,0.5,0.25],learning_rate_boundaries=[400000,500000],device_list=[0],clip_grad_norm=1.0', wdir='/home/timucin/Thesis/Tagger-master8')
File "/home/timucin/anaconda2/lib/python2.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 704, in runfile
execfile(filename, namespace)
File "/home/timucin/anaconda2/lib/python2.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 100, in execfile
builtins.execfile(filename, *where)
File "/home/timucin/Thesis/Tagger-master8/main.py", line 892, in <module>
train(parsed_args)
File "/home/timucin/Thesis/Tagger-master8/main.py", line 540, in train
experiment.train()
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 389, in train
saving_listeners=self._saving_listeners)
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 886, in _call_train
input_fn=input_fn, steps=steps, max_steps=max_steps, monitors=hooks)
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 525, in fit
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1095, in _train_model
_, loss = mon_sess.run([model_fn_ops.train_op, model_fn_ops.loss])
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 783, in __exit__
self._close_internal(exception_type)
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 821, in _close_internal
self._sess.close()
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1069, in close
self._sess.close()
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1229, in close
ignore_live_threads=True)
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 257, in _run
enqueue_callable()
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1257, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/timucin/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
FailedPreconditionError: /home/timucin/Thesis/Tagger-master8/train; Is a directory
[[{{node input_queues/examples_queue/parallel_read/ReaderReadV2}} = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](input_queues/examples_queue/parallel_read/TFRecordReaderV2, input_queues/examples_queue/parallel_read/filenames)]]
