{"id":386,"date":"2007-12-05T08:57:57","date_gmt":"2007-12-05T07:57:57","guid":{"rendered":"http:\/\/wp.devco.net\/?p=386"},"modified":"2009-10-09T14:09:36","modified_gmt":"2009-10-09T13:09:36","slug":"xen_no_space_left_on_device_sillyness","status":"publish","type":"post","link":"https:\/\/www.devco.net\/archives\/2007\/12\/05\/xen_no_space_left_on_device_sillyness.php","title":{"rendered":"XEN &#8216;No space left on device&#8217; sillyness."},"content":{"rendered":"<p>Yesterday while trying to get a i386 DomU going on my x86_64 Xen server I ran into some hassles with &#8216;No space left on device&#8217; errors.<br \/>\nAnyone who sees that would immediately go for the <i>df<\/i> command, but it would be futile in this instance.<br \/>\nWhat happens is that the xenstore &#8211; where it stores meta files state of the running VMs &#8211; gets corrupt,<br \/>\nYou can try and run &#8216;xenstore-control check&#8217; it will also give some b\/s answer kind of suggesting all is well, it&#8217;s not, check <i>\/var\/log\/messages<\/i> and you&#8217;ll see stuff like:<\/p>\n<blockquote>\n<pre>\nxenstored: corruption detected by connection 0: \\\nerr No such file or directory: Write  failed\nxenstored: clean_store: '\/local\/domain\/0\/backend\/vbd\/16\/51712\/sectors' is orphaned!\n<\/pre>\n<\/blockquote>\n<p>At this point you&#8217;re pretty much screwed, try and reboot and xend won&#8217;t even run, no VM&#8217;s will start.<br \/>\nFixing it is pretty easy in the end once you&#8217;ve done tons of Googling and found the 2 year old bug in the Xen bugtracker about this exact problem &#8211; complete with the xen guys trying to close it in a routine &#8216;cleanup of tickets&#8217; rather than actually fixing the bug.<br \/>\nFirst, shut down all things Xen, if you can even boot from a non Xen kernel.  Once you&#8217;re sure its all down just delete <i>\/var\/lib\/xenstored\/tdb*<\/i> and reboot, it should all be fine after that.<br \/>\nYou must be sure you don&#8217;t have xenstored running while doing this, else it will write its in-memory corrupted state back to disk when you reboot and it will look like your fix didn&#8217;t work.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Yesterday while trying to get a i386 DomU going on my x86_64 Xen server I ran into some hassles with &#8216;No space left on device&#8217; errors. Anyone who sees that would immediately go for the df command, but it would be futile in this instance. What happens is that the xenstore &#8211; where it stores [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[6],"tags":[19,48,47],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/386"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=386"}],"version-history":[{"count":1,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/386\/revisions"}],"predecessor-version":[{"id":553,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/386\/revisions\/553"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=386"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=386"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=386"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}